[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887385903558-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 16:43:06 dist-test-slave-wvz7 krb5kdc[395](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 16:43:06 dist-test-slave-wvz7 krb5kdc[395](info): set up 2 sockets
Jun 19 16:43:06 dist-test-slave-wvz7 krb5kdc[395](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260619 16:43:06.942127 388 mini_kdc.cc:121] Time spent starting KDC: real 0.956s user 0.000s sys 0.014s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887385903558-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887385903558-388-0/krb5kdc/test-user.keytab.
Jun 19 16:43:07 dist-test-slave-wvz7 krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887387, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T16:43:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T16:43:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887385903558-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887385903558-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260619 16:43:07.095592 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu
/tmp/dist-test-taskEs2EH8/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887385903558-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887385903558-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887385903558-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887385903558-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39333
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887385903558-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41595
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:39333
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887385903558-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887385903558-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887385903558-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:43:07.407634 408 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:43:07.407912 408 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:43:07.407996 408 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:43:07.415081 408 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:43:07.415192 408 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:43:07.415238 408 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:43:07.415293 408 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:43:07.415335 408 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:43:07.425410 408 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41595
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887385903558-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887385903558-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:39333
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887385903558-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39333
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887385903558-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887385903558-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:43:07.426728 408 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:43:07.427999 408 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:43:07.439152 413 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:43:07.439317 414 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:43:07.439476 416 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:43:07.440129 408 server_base.cc:1034] running on GCE node
I20260619 16:43:07.440831 408 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:43:07.442230 408 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:43:07.443461 408 hybrid_clock.cc:648] HybridClock initialized: now 1781887387443372 us; error 72 us; skew 500 ppm
I20260619 16:43:07.443786 408 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:43:07 dist-test-slave-wvz7 krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887387, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:43:07.446784 408 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260619 16:43:07.448298 408 webserver.cc:460] Webserver started at http://127.0.97.62:33333/ using document root <none> and password file <none>
I20260619 16:43:07.448943 408 fs_manager.cc:359] Metadata directory not provided
I20260619 16:43:07.449046 408 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:43:07.449292 408 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:43:07.452234 408 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887385903558-388-0/minicluster-data/master-0/data/instance:
uuid: "0f8e0454f03a42bb9e998c1e068c1fb9"
format_stamp: "Formatted at 2026-06-19 16:43:07 on dist-test-slave-wvz7"
server_key: "bb97eedba0a42733c68bf3ffb7abfc44"
server_key_iv: "9c0c666310d9881db2d6bbd6d8b22d3f"
server_key_version: "encryptionkey@0"
I20260619 16:43:07.452903 408 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887385903558-388-0/minicluster-data/master-0/wal/instance:
uuid: "0f8e0454f03a42bb9e998c1e068c1fb9"
format_stamp: "Formatted at 2026-06-19 16:43:07 on dist-test-slave-wvz7"
server_key: "bb97eedba0a42733c68bf3ffb7abfc44"
server_key_iv: "9c0c666310d9881db2d6bbd6d8b22d3f"
server_key_version: "encryptionkey@0"
I20260619 16:43:07.458128 408 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260619 16:43:07.462097 423 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:43:07.463460 408 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 16:43:07.463613 408 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887385903558-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887385903558-388-0/minicluster-data/master-0/wal
uuid: "0f8e0454f03a42bb9e998c1e068c1fb9"
format_stamp: "Formatted at 2026-06-19 16:43:07 on dist-test-slave-wvz7"
server_key: "bb97eedba0a42733c68bf3ffb7abfc44"
server_key_iv: "9c0c666310d9881db2d6bbd6d8b22d3f"
server_key_version: "encryptionkey@0"
I20260619 16:43:07.463789 408 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887385903558-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887385903558-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887385903558-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:43:07.475417 408 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:43:07.479526 408 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:43:07.479792 408 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:43:07.498580 408 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:39333
I20260619 16:43:07.498613 474 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:39333 every 8 connection(s)
I20260619 16:43:07.499907 408 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887385903558-388-0/minicluster-data/master-0/data/info.pb
I20260619 16:43:07.503584 475 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:43:07.510123 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu as pid 408
I20260619 16:43:07.510447 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887385903558-388-0/minicluster-data/master-0/wal/instance
I20260619 16:43:07.512909 388 external_mini_cluster.cc:1428] Setting key 91bdc4f18a8e0d19eca1d9d59d81d66e
I20260619 16:43:07.515733 475 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0f8e0454f03a42bb9e998c1e068c1fb9: Bootstrap starting.
I20260619 16:43:07.519851 475 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0f8e0454f03a42bb9e998c1e068c1fb9: Neither blocks nor log segments found. Creating new log.
I20260619 16:43:07.521152 475 log.cc:824] T 00000000000000000000000000000000 P 0f8e0454f03a42bb9e998c1e068c1fb9: Log is configured to *not* fsync() on all Append() calls
I20260619 16:43:07.524794 475 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0f8e0454f03a42bb9e998c1e068c1fb9: No bootstrap required, opened a new log
I20260619 16:43:07.534898 475 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0f8e0454f03a42bb9e998c1e068c1fb9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f8e0454f03a42bb9e998c1e068c1fb9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39333 } }
I20260619 16:43:07.535329 475 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0f8e0454f03a42bb9e998c1e068c1fb9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:43:07.535460 475 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0f8e0454f03a42bb9e998c1e068c1fb9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f8e0454f03a42bb9e998c1e068c1fb9, State: Initialized, Role: FOLLOWER
I20260619 16:43:07.536139 475 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0f8e0454f03a42bb9e998c1e068c1fb9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f8e0454f03a42bb9e998c1e068c1fb9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39333 } }
I20260619 16:43:07.536409 475 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0f8e0454f03a42bb9e998c1e068c1fb9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:43:07.536535 475 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0f8e0454f03a42bb9e998c1e068c1fb9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:43:07.536691 475 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0f8e0454f03a42bb9e998c1e068c1fb9 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:43:07.539922 475 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0f8e0454f03a42bb9e998c1e068c1fb9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f8e0454f03a42bb9e998c1e068c1fb9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39333 } }
I20260619 16:43:07.540393 475 leader_election.cc:302] T 00000000000000000000000000000000 P 0f8e0454f03a42bb9e998c1e068c1fb9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0f8e0454f03a42bb9e998c1e068c1fb9; no voters:
I20260619 16:43:07.540972 475 leader_election.cc:288] T 00000000000000000000000000000000 P 0f8e0454f03a42bb9e998c1e068c1fb9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 16:43:07.541040 480 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0f8e0454f03a42bb9e998c1e068c1fb9 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:43:07.542223 480 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0f8e0454f03a42bb9e998c1e068c1fb9 [term 1 LEADER]: Becoming Leader. State: Replica: 0f8e0454f03a42bb9e998c1e068c1fb9, State: Running, Role: LEADER
I20260619 16:43:07.542709 480 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0f8e0454f03a42bb9e998c1e068c1fb9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f8e0454f03a42bb9e998c1e068c1fb9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39333 } }
I20260619 16:43:07.543121 475 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0f8e0454f03a42bb9e998c1e068c1fb9 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:43:07.545158 482 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0f8e0454f03a42bb9e998c1e068c1fb9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0f8e0454f03a42bb9e998c1e068c1fb9. Latest consensus state: current_term: 1 leader_uuid: "0f8e0454f03a42bb9e998c1e068c1fb9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f8e0454f03a42bb9e998c1e068c1fb9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39333 } } }
I20260619 16:43:07.545186 481 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0f8e0454f03a42bb9e998c1e068c1fb9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0f8e0454f03a42bb9e998c1e068c1fb9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f8e0454f03a42bb9e998c1e068c1fb9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39333 } } }
I20260619 16:43:07.546043 481 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0f8e0454f03a42bb9e998c1e068c1fb9 [sys.catalog]: This master's current role is: LEADER
I20260619 16:43:07.546073 482 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0f8e0454f03a42bb9e998c1e068c1fb9 [sys.catalog]: This master's current role is: LEADER
Jun 19 16:43:07 dist-test-slave-wvz7 krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887387, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 16:43:07.562228 486 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:43:07.563210 477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:07.517506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45806 (local address 127.0.97.62:39333)
0619 16:43:07.518441 (+ 935us) server_negotiation.cc:206] Beginning negotiation
0619 16:43:07.518457 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0619 16:43:07.519812 (+ 1355us) server_negotiation.cc:407] Connection header received
0619 16:43:07.534406 (+ 14594us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:43:07.534432 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:43:07.534847 (+ 415us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:43:07.535192 (+ 345us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:43:07.539156 (+ 3964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:07.540572 (+ 1416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:07.542979 (+ 2407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:07.544011 (+ 1032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:07.544169 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:43:07.551728 (+ 7559us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:43:07.551740 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:43:07.551762 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:43:07.551817 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:43:07.556893 (+ 5076us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:43:07.558049 (+ 1156us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:43:07.558060 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:43:07.558070 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:43:07.558309 (+ 239us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:43:07.558888 (+ 579us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:43:07.558895 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:43:07.558899 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:43:07.559609 (+ 710us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:43:07.559898 (+ 289us) server_negotiation.cc:1035] Waiting for connection context
0619 16:43:07.561258 (+ 1360us) server_negotiation.cc:299] Negotiation successful
0619 16:43:07.561705 (+ 447us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":686,"thread_start_us":430,"threads_started":1}
I20260619 16:43:07.568514 486 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:43:07.576984 486 catalog_manager.cc:1347] Generated new cluster ID: 60dab138630d4207a915ece23ae671fb
I20260619 16:43:07.577095 486 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:43:07.604092 486 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 16:43:07.605782 486 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:43:07.615991 486 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0f8e0454f03a42bb9e998c1e068c1fb9: Generated new TSK 0
I20260619 16:43:07.616750 486 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887385903558-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887385903558-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260619 16:43:07.659922 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu
/tmp/dist-test-taskEs2EH8/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887385903558-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887385903558-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887385903558-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887385903558-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39333
--keytab_file=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887385903558-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41595
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887385903558-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887385903558-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887385903558-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:43:07.985720 501 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:43:07.986017 501 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:43:07.986155 501 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:43:07.993213 501 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:43:07.993338 501 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:43:07.993494 501 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260619 16:43:08.003432 501 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41595
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887385903558-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887385903558-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887385903558-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887385903558-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39333
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887385903558-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:43:08.004664 501 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:43:08.005963 501 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:43:08.017980 507 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:43:08.018220 509 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:43:08.018688 506 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:43:08.019116 501 server_base.cc:1034] running on GCE node
I20260619 16:43:08.019650 501 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:43:08.020537 501 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:43:08.021755 501 hybrid_clock.cc:648] HybridClock initialized: now 1781887388021678 us; error 44 us; skew 500 ppm
I20260619 16:43:08.022054 501 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:43:08 dist-test-slave-wvz7 krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887388, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:43:08.025278 501 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260619 16:43:08.026746 501 webserver.cc:460] Webserver started at http://127.0.97.1:37097/ using document root <none> and password file <none>
I20260619 16:43:08.027415 501 fs_manager.cc:359] Metadata directory not provided
I20260619 16:43:08.027526 501 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:43:08.027814 501 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:43:08.030649 501 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887385903558-388-0/minicluster-data/ts-0/data/instance:
uuid: "c23bac2aa8fa44d9b870ffe8111f3b25"
format_stamp: "Formatted at 2026-06-19 16:43:08 on dist-test-slave-wvz7"
server_key: "522bb351c07a9457ff998c2940743715"
server_key_iv: "f29c6b3e4b7c4505630d3e35a3ebe769"
server_key_version: "encryptionkey@0"
I20260619 16:43:08.031388 501 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887385903558-388-0/minicluster-data/ts-0/wal/instance:
uuid: "c23bac2aa8fa44d9b870ffe8111f3b25"
format_stamp: "Formatted at 2026-06-19 16:43:08 on dist-test-slave-wvz7"
server_key: "522bb351c07a9457ff998c2940743715"
server_key_iv: "f29c6b3e4b7c4505630d3e35a3ebe769"
server_key_version: "encryptionkey@0"
I20260619 16:43:08.036550 501 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260619 16:43:08.039901 516 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:43:08.041311 501 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 16:43:08.041460 501 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887385903558-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887385903558-388-0/minicluster-data/ts-0/wal
uuid: "c23bac2aa8fa44d9b870ffe8111f3b25"
format_stamp: "Formatted at 2026-06-19 16:43:08 on dist-test-slave-wvz7"
server_key: "522bb351c07a9457ff998c2940743715"
server_key_iv: "f29c6b3e4b7c4505630d3e35a3ebe769"
server_key_version: "encryptionkey@0"
I20260619 16:43:08.041646 501 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887385903558-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887385903558-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887385903558-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:43:08.083321 501 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:43:08.087580 501 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:43:08.087929 501 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:43:08.088900 501 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:43:08.090198 501 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:43:08.090273 501 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:43:08.090386 501 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:43:08.090437 501 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:43:08.122884 501 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:38809
I20260619 16:43:08.122931 628 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:38809 every 8 connection(s)
I20260619 16:43:08.124050 501 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887385903558-388-0/minicluster-data/ts-0/data/info.pb
I20260619 16:43:08.125841 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu as pid 501
I20260619 16:43:08.126030 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887385903558-388-0/minicluster-data/ts-0/wal/instance
I20260619 16:43:08.126641 388 external_mini_cluster.cc:1428] Setting key 7801997bea50be7dd5b3a6036a5e1d3f
Jun 19 16:43:08 dist-test-slave-wvz7 krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887388, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260619 16:43:08.144332 631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:08.126516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:33733 (local address 127.0.97.62:39333)
0619 16:43:08.126954 (+ 438us) server_negotiation.cc:206] Beginning negotiation
0619 16:43:08.126959 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:43:08.128241 (+ 1282us) server_negotiation.cc:407] Connection header received
0619 16:43:08.130064 (+ 1823us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:43:08.130070 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:43:08.130150 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:43:08.130235 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:43:08.131639 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:08.132370 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:08.133972 (+ 1602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:08.134569 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:08.134702 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:43:08.138190 (+ 3488us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:43:08.138198 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:43:08.138203 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:43:08.138242 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:43:08.141360 (+ 3118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:43:08.142158 (+ 798us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:43:08.142163 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:43:08.142165 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:43:08.142290 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:43:08.142689 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:43:08.142692 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:43:08.142695 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:43:08.142980 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:43:08.143103 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0619 16:43:08.143807 (+ 704us) server_negotiation.cc:299] Negotiation successful
0619 16:43:08.143996 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":320,"thread_start_us":206,"threads_started":1}
I20260619 16:43:08.145781 632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:08.127039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39333 (local address 127.0.97.1:33733)
0619 16:43:08.128111 (+ 1072us) negotiation.cc:107] Waiting for socket to connect
0619 16:43:08.128148 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0619 16:43:08.129830 (+ 1682us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:43:08.130461 (+ 631us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:43:08.130469 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:43:08.130942 (+ 473us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:43:08.131484 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:43:08.131502 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:43:08.132526 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:43:08.132531 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:43:08.133833 (+ 1302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:43:08.133849 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:43:08.134796 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:43:08.134802 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:43:08.135018 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:43:08.135653 (+ 635us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:43:08.135675 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:43:08.137980 (+ 2305us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:43:08.141508 (+ 3528us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:43:08.141523 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:43:08.141536 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:43:08.142006 (+ 470us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:43:08.142402 (+ 396us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:43:08.142405 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:43:08.142408 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:43:08.142592 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:43:08.143076 (+ 484us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:43:08.143082 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:43:08.143398 (+ 316us) client_negotiation.cc:769] Sending connection context
0619 16:43:08.143683 (+ 285us) client_negotiation.cc:240] Negotiation successful
0619 16:43:08.144231 (+ 548us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":860,"thread_start_us":714,"threads_started":1}
I20260619 16:43:08.147337 629 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39333
I20260619 16:43:08.147609 629 heartbeater.cc:464] Registering TS with master...
I20260619 16:43:08.148377 629 heartbeater.cc:511] Master 127.0.97.62:39333 requested a full tablet report, sending...
I20260619 16:43:08.150686 440 ts_manager.cc:194] Registered new tserver with Master: c23bac2aa8fa44d9b870ffe8111f3b25 (127.0.97.1:38809)
I20260619 16:43:08.152565 440 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:33733
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887385903558-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887385903558-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260619 16:43:08.164433 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu
/tmp/dist-test-taskEs2EH8/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887385903558-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887385903558-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887385903558-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887385903558-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39333
--keytab_file=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887385903558-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41595
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887385903558-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887385903558-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887385903558-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:43:08.467761 635 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:43:08.468039 635 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:43:08.468142 635 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:43:08.475317 635 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:43:08.475446 635 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:43:08.475602 635 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260619 16:43:08.485288 635 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41595
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887385903558-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887385903558-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887385903558-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887385903558-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39333
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887385903558-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:43:08.486533 635 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:43:08.487794 635 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:43:08.500557 641 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:43:08.500658 640 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:43:08.501037 643 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:43:08.502418 635 server_base.cc:1034] running on GCE node
I20260619 16:43:08.502951 635 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:43:08.503847 635 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:43:08.505033 635 hybrid_clock.cc:648] HybridClock initialized: now 1781887388504979 us; error 51 us; skew 500 ppm
I20260619 16:43:08.505326 635 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:43:08 dist-test-slave-wvz7 krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887388, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:43:08.508179 635 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260619 16:43:08.509716 635 webserver.cc:460] Webserver started at http://127.0.97.2:41757/ using document root <none> and password file <none>
I20260619 16:43:08.510457 635 fs_manager.cc:359] Metadata directory not provided
I20260619 16:43:08.510552 635 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:43:08.510790 635 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:43:08.514396 635 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887385903558-388-0/minicluster-data/ts-1/data/instance:
uuid: "9957e27912834ae89cd09c72f24a9a21"
format_stamp: "Formatted at 2026-06-19 16:43:08 on dist-test-slave-wvz7"
server_key: "53083d1dbdb23bff776e77c4256bb306"
server_key_iv: "9158b0a8367eaccfa62c1ff29044158a"
server_key_version: "encryptionkey@0"
I20260619 16:43:08.515408 635 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887385903558-388-0/minicluster-data/ts-1/wal/instance:
uuid: "9957e27912834ae89cd09c72f24a9a21"
format_stamp: "Formatted at 2026-06-19 16:43:08 on dist-test-slave-wvz7"
server_key: "53083d1dbdb23bff776e77c4256bb306"
server_key_iv: "9158b0a8367eaccfa62c1ff29044158a"
server_key_version: "encryptionkey@0"
I20260619 16:43:08.521569 635 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260619 16:43:08.525556 650 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:43:08.526803 635 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 16:43:08.526934 635 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887385903558-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887385903558-388-0/minicluster-data/ts-1/wal
uuid: "9957e27912834ae89cd09c72f24a9a21"
format_stamp: "Formatted at 2026-06-19 16:43:08 on dist-test-slave-wvz7"
server_key: "53083d1dbdb23bff776e77c4256bb306"
server_key_iv: "9158b0a8367eaccfa62c1ff29044158a"
server_key_version: "encryptionkey@0"
I20260619 16:43:08.527109 635 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887385903558-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887385903558-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887385903558-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:43:08.550585 635 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:43:08.554548 635 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:43:08.554797 635 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:43:08.555660 635 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:43:08.556869 635 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:43:08.556941 635 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:43:08.557447 635 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:43:08.557507 635 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:43:08.588320 635 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:35367
I20260619 16:43:08.588359 762 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:35367 every 8 connection(s)
I20260619 16:43:08.589462 635 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887385903558-388-0/minicluster-data/ts-1/data/info.pb
I20260619 16:43:08.598384 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu as pid 635
I20260619 16:43:08.598568 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887385903558-388-0/minicluster-data/ts-1/wal/instance
I20260619 16:43:08.599073 388 external_mini_cluster.cc:1428] Setting key 79221737979811d55d445dee0f41992c
Jun 19 16:43:08 dist-test-slave-wvz7 krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887388, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 16:43:08.609266 631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:08.591828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:43093 (local address 127.0.97.62:39333)
0619 16:43:08.591993 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0619 16:43:08.591998 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:43:08.593030 (+ 1032us) server_negotiation.cc:407] Connection header received
0619 16:43:08.594765 (+ 1735us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:43:08.594769 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:43:08.594831 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:43:08.594914 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:43:08.596221 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:08.596910 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:08.599262 (+ 2352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:08.599800 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:08.599924 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:43:08.603850 (+ 3926us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:43:08.603855 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:43:08.603858 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:43:08.603890 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:43:08.606537 (+ 2647us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:43:08.607159 (+ 622us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:43:08.607164 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:43:08.607169 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:43:08.607306 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:43:08.607661 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:43:08.607665 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:43:08.607669 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:43:08.608004 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:43:08.608134 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0619 16:43:08.608727 (+ 593us) server_negotiation.cc:299] Negotiation successful
0619 16:43:08.608934 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260619 16:43:08.610085 765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:08.592272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39333 (local address 127.0.97.2:43093)
0619 16:43:08.592890 (+ 618us) negotiation.cc:107] Waiting for socket to connect
0619 16:43:08.592925 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0619 16:43:08.594528 (+ 1603us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:43:08.595074 (+ 546us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:43:08.595081 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:43:08.595539 (+ 458us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:43:08.596080 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:43:08.596105 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:43:08.597056 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:43:08.597059 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:43:08.599062 (+ 2003us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:43:08.599082 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:43:08.600020 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:43:08.600027 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:43:08.600265 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:43:08.600956 (+ 691us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:43:08.600978 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:43:08.603618 (+ 2640us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:43:08.606654 (+ 3036us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:43:08.606660 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:43:08.606672 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:43:08.607048 (+ 376us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:43:08.607397 (+ 349us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:43:08.607400 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:43:08.607402 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:43:08.607558 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:43:08.608112 (+ 554us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:43:08.608117 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:43:08.608400 (+ 283us) client_negotiation.cc:769] Sending connection context
0619 16:43:08.608624 (+ 224us) client_negotiation.cc:240] Negotiation successful
0619 16:43:08.608985 (+ 361us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":409,"thread_start_us":235,"threads_started":1}
I20260619 16:43:08.611375 763 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39333
I20260619 16:43:08.611750 763 heartbeater.cc:464] Registering TS with master...
I20260619 16:43:08.612596 763 heartbeater.cc:511] Master 127.0.97.62:39333 requested a full tablet report, sending...
I20260619 16:43:08.614070 440 ts_manager.cc:194] Registered new tserver with Master: 9957e27912834ae89cd09c72f24a9a21 (127.0.97.2:35367)
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260619 16:43:08.614835 440 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:43093
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887385903558-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887385903558-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260619 16:43:08.634164 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu
/tmp/dist-test-taskEs2EH8/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887385903558-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887385903558-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887385903558-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887385903558-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39333
--keytab_file=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887385903558-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41595
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887385903558-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887385903558-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887385903558-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:43:08.943229 768 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:43:08.943485 768 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:43:08.943614 768 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:43:08.950773 768 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:43:08.950870 768 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:43:08.951026 768 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260619 16:43:08.961201 768 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41595
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887385903558-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887385903558-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887385903558-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887385903558-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39333
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887385903558-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:43:08.962507 768 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:43:08.963801 768 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:43:08.976933 773 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:43:08.976861 774 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:43:08.977429 776 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:43:08.977715 768 server_base.cc:1034] running on GCE node
I20260619 16:43:08.978355 768 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:43:08.979264 768 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:43:08.980470 768 hybrid_clock.cc:648] HybridClock initialized: now 1781887388980415 us; error 42 us; skew 500 ppm
I20260619 16:43:08.980805 768 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:43:08 dist-test-slave-wvz7 krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887388, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:43:08.986866 768 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260619 16:43:08.988286 768 webserver.cc:460] Webserver started at http://127.0.97.3:33655/ using document root <none> and password file <none>
I20260619 16:43:08.988930 768 fs_manager.cc:359] Metadata directory not provided
I20260619 16:43:08.989037 768 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:43:08.989300 768 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:43:08.992041 768 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887385903558-388-0/minicluster-data/ts-2/data/instance:
uuid: "9ab1624e813540d0b6396e942f68fd53"
format_stamp: "Formatted at 2026-06-19 16:43:08 on dist-test-slave-wvz7"
server_key: "0a4e7a7aea77a5a894f1d6bee0a0abd3"
server_key_iv: "7ab45cd4670f567866eeb2caa14c030d"
server_key_version: "encryptionkey@0"
I20260619 16:43:08.993166 768 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887385903558-388-0/minicluster-data/ts-2/wal/instance:
uuid: "9ab1624e813540d0b6396e942f68fd53"
format_stamp: "Formatted at 2026-06-19 16:43:08 on dist-test-slave-wvz7"
server_key: "0a4e7a7aea77a5a894f1d6bee0a0abd3"
server_key_iv: "7ab45cd4670f567866eeb2caa14c030d"
server_key_version: "encryptionkey@0"
I20260619 16:43:08.998026 768 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.002s
I20260619 16:43:09.001212 783 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:43:09.002460 768 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 16:43:09.002599 768 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887385903558-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887385903558-388-0/minicluster-data/ts-2/wal
uuid: "9ab1624e813540d0b6396e942f68fd53"
format_stamp: "Formatted at 2026-06-19 16:43:08 on dist-test-slave-wvz7"
server_key: "0a4e7a7aea77a5a894f1d6bee0a0abd3"
server_key_iv: "7ab45cd4670f567866eeb2caa14c030d"
server_key_version: "encryptionkey@0"
I20260619 16:43:09.002765 768 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887385903558-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887385903558-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887385903558-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:43:09.024591 768 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:43:09.028584 768 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:43:09.028829 768 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:43:09.030148 768 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:43:09.031466 768 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:43:09.031536 768 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:43:09.031643 768 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:43:09.031690 768 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:43:09.062532 768 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:42479
I20260619 16:43:09.062567 895 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:42479 every 8 connection(s)
I20260619 16:43:09.063738 768 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887385903558-388-0/minicluster-data/ts-2/data/info.pb
I20260619 16:43:09.067485 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu as pid 768
I20260619 16:43:09.067683 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887385903558-388-0/minicluster-data/ts-2/wal/instance
I20260619 16:43:09.068192 388 external_mini_cluster.cc:1428] Setting key 20645050c05d8f82bedbfc94ca8a81f9
Jun 19 16:43:09 dist-test-slave-wvz7 krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887388, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 16:43:09.083138 631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:09.065981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:53921 (local address 127.0.97.62:39333)
0619 16:43:09.066140 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0619 16:43:09.066144 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:43:09.067283 (+ 1139us) server_negotiation.cc:407] Connection header received
0619 16:43:09.069977 (+ 2694us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:43:09.069981 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:43:09.070042 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:43:09.070125 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:43:09.071800 (+ 1675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:09.072490 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:09.074007 (+ 1517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:09.074444 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:09.074551 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:43:09.077653 (+ 3102us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:43:09.077658 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:43:09.077661 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:43:09.077695 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:43:09.080148 (+ 2453us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:43:09.080769 (+ 621us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:43:09.080773 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:43:09.080775 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:43:09.080888 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:43:09.081391 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:43:09.081394 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:43:09.081396 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:43:09.081647 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:43:09.081782 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0619 16:43:09.082587 (+ 805us) server_negotiation.cc:299] Negotiation successful
0619 16:43:09.082792 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260619 16:43:09.084055 898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:09.066431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39333 (local address 127.0.97.3:53921)
0619 16:43:09.067110 (+ 679us) negotiation.cc:107] Waiting for socket to connect
0619 16:43:09.067163 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0619 16:43:09.069684 (+ 2521us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:43:09.070372 (+ 688us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:43:09.070381 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:43:09.071018 (+ 637us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:43:09.071631 (+ 613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:43:09.071655 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:43:09.072611 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:43:09.072615 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:43:09.073854 (+ 1239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:43:09.073866 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:43:09.074649 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:43:09.074654 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:43:09.074845 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:43:09.075452 (+ 607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:43:09.075474 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:43:09.077461 (+ 1987us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:43:09.080270 (+ 2809us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:43:09.080276 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:43:09.080286 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:43:09.080659 (+ 373us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:43:09.081095 (+ 436us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:43:09.081099 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:43:09.081101 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:43:09.081293 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:43:09.081818 (+ 525us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:43:09.081825 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:43:09.082196 (+ 371us) client_negotiation.cc:769] Sending connection context
0619 16:43:09.082432 (+ 236us) client_negotiation.cc:240] Negotiation successful
0619 16:43:09.082902 (+ 470us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":408,"thread_start_us":237,"threads_started":1}
I20260619 16:43:09.085232 896 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39333
I20260619 16:43:09.085494 896 heartbeater.cc:464] Registering TS with master...
I20260619 16:43:09.086169 896 heartbeater.cc:511] Master 127.0.97.62:39333 requested a full tablet report, sending...
I20260619 16:43:09.087409 440 ts_manager.cc:194] Registered new tserver with Master: 9ab1624e813540d0b6396e942f68fd53 (127.0.97.3:42479)
I20260619 16:43:09.088141 440 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:53921
I20260619 16:43:09.095520 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 19 16:43:09 dist-test-slave-wvz7 krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887389, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 19 16:43:09 dist-test-slave-wvz7 krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887389, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260619 16:43:09.136121 902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:09.121411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52860 (local address 127.0.97.1:38809)
0619 16:43:09.121890 (+ 479us) server_negotiation.cc:206] Beginning negotiation
0619 16:43:09.121900 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 16:43:09.122364 (+ 464us) server_negotiation.cc:407] Connection header received
0619 16:43:09.122516 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:43:09.122524 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:43:09.122799 (+ 275us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:43:09.122973 (+ 174us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:43:09.124197 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:09.125771 (+ 1574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:09.126963 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:09.127728 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:09.127819 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:43:09.131492 (+ 3673us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:43:09.131502 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:43:09.131515 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:43:09.131547 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:43:09.134001 (+ 2454us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:43:09.134537 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:43:09.134543 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:43:09.134545 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:43:09.134659 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:43:09.134942 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:43:09.134945 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:43:09.134947 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:43:09.135266 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:43:09.135470 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0619 16:43:09.135679 (+ 209us) server_negotiation.cc:299] Negotiation successful
0619 16:43:09.135854 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":331,"thread_start_us":212,"threads_started":1}
W20260619 16:43:09.137228 544 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52860
I20260619 16:43:09.156241 629 heartbeater.cc:503] Master 127.0.97.62:39333 was elected leader, sending a full tablet report...
Jun 19 16:43:09 dist-test-slave-wvz7 krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887389, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 19 16:43:09 dist-test-slave-wvz7 krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887389, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260619 16:43:09.175143 902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:09.163435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52870 (local address 127.0.97.1:38809)
0619 16:43:09.163701 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0619 16:43:09.163708 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:43:09.163784 (+ 76us) server_negotiation.cc:407] Connection header received
0619 16:43:09.163968 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:43:09.163972 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:43:09.164034 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:43:09.164118 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:43:09.165323 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:09.166007 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:09.167214 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:09.167612 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:09.167693 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:43:09.171174 (+ 3481us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:43:09.171179 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:43:09.171182 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:43:09.171211 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:43:09.173310 (+ 2099us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:43:09.173835 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:43:09.173838 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:43:09.173840 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:43:09.173953 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:43:09.174232 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:43:09.174235 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:43:09.174237 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:43:09.174451 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:43:09.174619 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0619 16:43:09.174794 (+ 175us) server_negotiation.cc:299] Negotiation successful
0619 16:43:09.174937 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115}
I20260619 16:43:09.176431 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu pid 501
I20260619 16:43:09.188251 902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:09.177264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52882 (local address 127.0.97.1:38809)
0619 16:43:09.177447 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0619 16:43:09.177453 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:43:09.177573 (+ 120us) server_negotiation.cc:407] Connection header received
0619 16:43:09.177672 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:43:09.177675 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:43:09.177726 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:43:09.177831 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:43:09.179419 (+ 1588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:09.180043 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:09.181329 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:09.181716 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:09.181831 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:43:09.183925 (+ 2094us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:43:09.183930 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:43:09.183933 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:43:09.183958 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:43:09.186224 (+ 2266us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:43:09.186727 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:43:09.186731 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:43:09.186733 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:43:09.186836 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:43:09.187111 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:43:09.187114 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:43:09.187116 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:43:09.187317 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:43:09.187462 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0619 16:43:09.187812 (+ 350us) server_negotiation.cc:299] Negotiation successful
0619 16:43:09.187945 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260619 16:43:09.189003 624 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:43:09.271734 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu with pid 501
I20260619 16:43:09.284317 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu pid 635
Jun 19 16:43:09 dist-test-slave-wvz7 krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887389, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260619 16:43:09.299124 910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:09.285079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50310 (local address 127.0.97.2:35367)
0619 16:43:09.285600 (+ 521us) server_negotiation.cc:206] Beginning negotiation
0619 16:43:09.285607 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:43:09.285630 (+ 23us) server_negotiation.cc:407] Connection header received
0619 16:43:09.285685 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:43:09.285692 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:43:09.285946 (+ 254us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:43:09.286097 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:43:09.287215 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:09.288221 (+ 1006us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:09.289410 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:09.290027 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:09.290156 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:43:09.293882 (+ 3726us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:43:09.293892 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:43:09.293906 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:43:09.293938 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:43:09.296456 (+ 2518us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:43:09.297078 (+ 622us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:43:09.297085 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:43:09.297087 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:43:09.297199 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:43:09.297577 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:43:09.297581 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:43:09.297585 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:43:09.297989 (+ 404us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:43:09.298283 (+ 294us) server_negotiation.cc:1035] Waiting for connection context
0619 16:43:09.298533 (+ 250us) server_negotiation.cc:299] Negotiation successful
0619 16:43:09.298776 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":392,"thread_start_us":260,"threads_started":1}
I20260619 16:43:09.300295 758 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:43:09.379608 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu with pid 635
I20260619 16:43:09.397171 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu pid 768
Jun 19 16:43:09 dist-test-slave-wvz7 krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887389, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260619 16:43:09.412451 912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:09.398192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41958 (local address 127.0.97.3:42479)
0619 16:43:09.398786 (+ 594us) server_negotiation.cc:206] Beginning negotiation
0619 16:43:09.398793 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:43:09.398821 (+ 28us) server_negotiation.cc:407] Connection header received
0619 16:43:09.398876 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:43:09.398882 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:43:09.399139 (+ 257us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:43:09.399289 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:43:09.400475 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:09.401491 (+ 1016us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:09.402821 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:09.403377 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:09.403488 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:43:09.407322 (+ 3834us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:43:09.407332 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:43:09.407346 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:43:09.407400 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:43:09.409901 (+ 2501us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:43:09.410540 (+ 639us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:43:09.410547 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:43:09.410551 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:43:09.410668 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:43:09.410993 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:43:09.410997 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:43:09.410999 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:43:09.411333 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:43:09.411601 (+ 268us) server_negotiation.cc:1035] Waiting for connection context
0619 16:43:09.411905 (+ 304us) server_negotiation.cc:299] Negotiation successful
0619 16:43:09.412158 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":459,"thread_start_us":322,"threads_started":1}
I20260619 16:43:09.413627 891 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:43:09.493032 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu with pid 768
I20260619 16:43:09.505299 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu pid 408
I20260619 16:43:09.506364 470 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:43:09.604692 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu with pid 408
2026-06-19T16:43:09Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (3663 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887385903558-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 16:43:09 dist-test-slave-wvz7 krb5kdc[917](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 16:43:09 dist-test-slave-wvz7 krb5kdc[917](info): set up 2 sockets
Jun 19 16:43:09 dist-test-slave-wvz7 krb5kdc[917](info): commencing operation
krb5kdc: starting...
W20260619 16:43:10.593653 388 mini_kdc.cc:121] Time spent starting KDC: real 0.953s user 0.001s sys 0.012s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887385903558-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887385903558-388-0/krb5kdc/test-user.keytab.
Jun 19 16:43:10 dist-test-slave-wvz7 krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887390, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T16:43:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T16:43:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887385903558-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887385903558-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260619 16:43:10.750149 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu
/tmp/dist-test-taskEs2EH8/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887385903558-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887385903558-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887385903558-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887385903558-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:43151
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887385903558-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37577
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:43151
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887385903558-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887385903558-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887385903558-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:43:11.050187 930 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:43:11.050477 930 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:43:11.050601 930 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:43:11.057688 930 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:43:11.057811 930 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:43:11.057889 930 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:43:11.057947 930 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:43:11.057991 930 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:43:11.067724 930 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37577
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887385903558-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887385903558-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:43151
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887385903558-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:43151
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887385903558-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887385903558-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:43:11.069005 930 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:43:11.070271 930 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:43:11.081301 938 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:43:11.081400 936 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:43:11.081379 935 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:43:11.082233 930 server_base.cc:1034] running on GCE node
I20260619 16:43:11.082907 930 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:43:11.084244 930 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:43:11.085457 930 hybrid_clock.cc:648] HybridClock initialized: now 1781887391085386 us; error 63 us; skew 500 ppm
I20260619 16:43:11.085829 930 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:43:11 dist-test-slave-wvz7 krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887391, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:43:11.088714 930 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260619 16:43:11.090185 930 webserver.cc:460] Webserver started at http://127.0.97.62:42127/ using document root <none> and password file <none>
I20260619 16:43:11.090829 930 fs_manager.cc:359] Metadata directory not provided
I20260619 16:43:11.090938 930 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:43:11.091215 930 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:43:11.094002 930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887385903558-388-0/minicluster-data/master-0/data/instance:
uuid: "0359fb8fc2774177b35e91191ff8e468"
format_stamp: "Formatted at 2026-06-19 16:43:11 on dist-test-slave-wvz7"
server_key: "e778a27c550983aabe07841f6eb7d0ce"
server_key_iv: "94e2c32fae12bef424159c7faf31c002"
server_key_version: "encryptionkey@0"
I20260619 16:43:11.094676 930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887385903558-388-0/minicluster-data/master-0/wal/instance:
uuid: "0359fb8fc2774177b35e91191ff8e468"
format_stamp: "Formatted at 2026-06-19 16:43:11 on dist-test-slave-wvz7"
server_key: "e778a27c550983aabe07841f6eb7d0ce"
server_key_iv: "94e2c32fae12bef424159c7faf31c002"
server_key_version: "encryptionkey@0"
I20260619 16:43:11.099459 930 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 16:43:11.102602 945 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:43:11.103850 930 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 16:43:11.104017 930 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887385903558-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887385903558-388-0/minicluster-data/master-0/wal
uuid: "0359fb8fc2774177b35e91191ff8e468"
format_stamp: "Formatted at 2026-06-19 16:43:11 on dist-test-slave-wvz7"
server_key: "e778a27c550983aabe07841f6eb7d0ce"
server_key_iv: "94e2c32fae12bef424159c7faf31c002"
server_key_version: "encryptionkey@0"
I20260619 16:43:11.104179 930 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887385903558-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887385903558-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887385903558-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:43:11.123049 930 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:43:11.126895 930 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:43:11.127151 930 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:43:11.143330 930 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:43151
I20260619 16:43:11.143402 996 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:43151 every 8 connection(s)
I20260619 16:43:11.144619 930 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887385903558-388-0/minicluster-data/master-0/data/info.pb
I20260619 16:43:11.148244 997 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:43:11.154783 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu as pid 930
I20260619 16:43:11.154999 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887385903558-388-0/minicluster-data/master-0/wal/instance
I20260619 16:43:11.155606 388 external_mini_cluster.cc:1428] Setting key cd5288567f23a980942dae35449dfae4
I20260619 16:43:11.158936 997 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0359fb8fc2774177b35e91191ff8e468: Bootstrap starting.
I20260619 16:43:11.162782 997 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0359fb8fc2774177b35e91191ff8e468: Neither blocks nor log segments found. Creating new log.
I20260619 16:43:11.164170 997 log.cc:824] T 00000000000000000000000000000000 P 0359fb8fc2774177b35e91191ff8e468: Log is configured to *not* fsync() on all Append() calls
I20260619 16:43:11.166921 997 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0359fb8fc2774177b35e91191ff8e468: No bootstrap required, opened a new log
Jun 19 16:43:11 dist-test-slave-wvz7 krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887390, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 16:43:11.175290 997 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0359fb8fc2774177b35e91191ff8e468 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0359fb8fc2774177b35e91191ff8e468" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43151 } }
I20260619 16:43:11.175673 997 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0359fb8fc2774177b35e91191ff8e468 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:43:11.175776 997 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0359fb8fc2774177b35e91191ff8e468 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0359fb8fc2774177b35e91191ff8e468, State: Initialized, Role: FOLLOWER
I20260619 16:43:11.176416 997 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0359fb8fc2774177b35e91191ff8e468 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0359fb8fc2774177b35e91191ff8e468" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43151 } }
I20260619 16:43:11.176653 997 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0359fb8fc2774177b35e91191ff8e468 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:43:11.176774 997 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0359fb8fc2774177b35e91191ff8e468 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:43:11.176934 997 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0359fb8fc2774177b35e91191ff8e468 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:43:11.179147 1000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:11.157422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39114 (local address 127.0.97.62:43151)
0619 16:43:11.158246 (+ 824us) server_negotiation.cc:206] Beginning negotiation
0619 16:43:11.158261 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 16:43:11.158311 (+ 50us) server_negotiation.cc:407] Connection header received
0619 16:43:11.160742 (+ 2431us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:43:11.160776 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:43:11.161277 (+ 501us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:43:11.161696 (+ 419us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:43:11.162917 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:11.164934 (+ 2017us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:11.166283 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:11.166985 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:11.167170 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:43:11.171489 (+ 4319us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:43:11.171498 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:43:11.171513 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:43:11.171562 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:43:11.174939 (+ 3377us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:43:11.175599 (+ 660us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:43:11.175606 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:43:11.175613 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:43:11.175815 (+ 202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:43:11.176277 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:43:11.176282 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:43:11.176284 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:43:11.176874 (+ 590us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:43:11.177144 (+ 270us) server_negotiation.cc:1035] Waiting for connection context
0619 16:43:11.177613 (+ 469us) server_negotiation.cc:299] Negotiation successful
0619 16:43:11.178100 (+ 487us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":595,"thread_start_us":364,"threads_started":1}
I20260619 16:43:11.179605 997 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0359fb8fc2774177b35e91191ff8e468 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0359fb8fc2774177b35e91191ff8e468" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43151 } }
I20260619 16:43:11.180151 997 leader_election.cc:302] T 00000000000000000000000000000000 P 0359fb8fc2774177b35e91191ff8e468 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0359fb8fc2774177b35e91191ff8e468; no voters:
I20260619 16:43:11.180750 997 leader_election.cc:288] T 00000000000000000000000000000000 P 0359fb8fc2774177b35e91191ff8e468 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 16:43:11.180831 1002 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0359fb8fc2774177b35e91191ff8e468 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:43:11.182809 997 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0359fb8fc2774177b35e91191ff8e468 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:43:11.187345 1002 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0359fb8fc2774177b35e91191ff8e468 [term 1 LEADER]: Becoming Leader. State: Replica: 0359fb8fc2774177b35e91191ff8e468, State: Running, Role: LEADER
I20260619 16:43:11.187812 1002 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0359fb8fc2774177b35e91191ff8e468 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0359fb8fc2774177b35e91191ff8e468" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43151 } }
I20260619 16:43:11.190245 1003 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0359fb8fc2774177b35e91191ff8e468 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0359fb8fc2774177b35e91191ff8e468" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0359fb8fc2774177b35e91191ff8e468" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43151 } } }
I20260619 16:43:11.190547 1003 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0359fb8fc2774177b35e91191ff8e468 [sys.catalog]: This master's current role is: LEADER
I20260619 16:43:11.191473 1015 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:43:11.191304 1004 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0359fb8fc2774177b35e91191ff8e468 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0359fb8fc2774177b35e91191ff8e468. Latest consensus state: current_term: 1 leader_uuid: "0359fb8fc2774177b35e91191ff8e468" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0359fb8fc2774177b35e91191ff8e468" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43151 } } }
I20260619 16:43:11.191706 1004 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0359fb8fc2774177b35e91191ff8e468 [sys.catalog]: This master's current role is: LEADER
I20260619 16:43:11.197353 1015 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:43:11.205895 1015 catalog_manager.cc:1347] Generated new cluster ID: a9ba5723d7874909b270cbab5bc1e19f
I20260619 16:43:11.206007 1015 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:43:11.217348 1015 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 16:43:11.219117 1015 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:43:11.236161 1015 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0359fb8fc2774177b35e91191ff8e468: Generated new TSK 0
I20260619 16:43:11.237229 1015 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887385903558-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887385903558-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260619 16:43:11.285513 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu
/tmp/dist-test-taskEs2EH8/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887385903558-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887385903558-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887385903558-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887385903558-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43151
--keytab_file=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887385903558-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37577
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887385903558-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887385903558-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887385903558-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:43:11.587407 1023 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:43:11.587677 1023 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:43:11.587811 1023 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:43:11.595077 1023 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:43:11.595182 1023 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:43:11.595322 1023 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260619 16:43:11.604764 1023 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37577
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887385903558-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887385903558-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887385903558-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887385903558-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43151
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887385903558-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:43:11.606070 1023 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:43:11.607582 1023 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:43:11.620272 1028 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:43:11.620222 1029 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:43:11.620195 1031 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:43:11.620700 1023 server_base.cc:1034] running on GCE node
I20260619 16:43:11.621433 1023 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:43:11.622339 1023 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:43:11.623560 1023 hybrid_clock.cc:648] HybridClock initialized: now 1781887391623514 us; error 47 us; skew 500 ppm
I20260619 16:43:11.623858 1023 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:43:11 dist-test-slave-wvz7 krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887391, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:43:11.626781 1023 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260619 16:43:11.628104 1023 webserver.cc:460] Webserver started at http://127.0.97.1:35287/ using document root <none> and password file <none>
I20260619 16:43:11.628749 1023 fs_manager.cc:359] Metadata directory not provided
I20260619 16:43:11.628859 1023 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:43:11.629141 1023 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:43:11.632010 1023 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887385903558-388-0/minicluster-data/ts-0/data/instance:
uuid: "35efb5d26f4f4307b2f5723abb6749ee"
format_stamp: "Formatted at 2026-06-19 16:43:11 on dist-test-slave-wvz7"
server_key: "f4a0af99bca08d1307b5af9cc8b1531b"
server_key_iv: "d31f9ee2e3897f2bac3dac9b1b184c54"
server_key_version: "encryptionkey@0"
I20260619 16:43:11.632737 1023 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887385903558-388-0/minicluster-data/ts-0/wal/instance:
uuid: "35efb5d26f4f4307b2f5723abb6749ee"
format_stamp: "Formatted at 2026-06-19 16:43:11 on dist-test-slave-wvz7"
server_key: "f4a0af99bca08d1307b5af9cc8b1531b"
server_key_iv: "d31f9ee2e3897f2bac3dac9b1b184c54"
server_key_version: "encryptionkey@0"
I20260619 16:43:11.637576 1023 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.003s
I20260619 16:43:11.640821 1038 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:43:11.642104 1023 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 16:43:11.642242 1023 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887385903558-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887385903558-388-0/minicluster-data/ts-0/wal
uuid: "35efb5d26f4f4307b2f5723abb6749ee"
format_stamp: "Formatted at 2026-06-19 16:43:11 on dist-test-slave-wvz7"
server_key: "f4a0af99bca08d1307b5af9cc8b1531b"
server_key_iv: "d31f9ee2e3897f2bac3dac9b1b184c54"
server_key_version: "encryptionkey@0"
I20260619 16:43:11.642413 1023 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887385903558-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887385903558-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887385903558-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:43:11.655328 1023 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:43:11.658872 1023 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:43:11.659108 1023 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:43:11.659958 1023 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:43:11.661130 1023 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:43:11.661201 1023 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:43:11.661307 1023 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:43:11.661370 1023 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:43:11.689970 1023 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:43015
I20260619 16:43:11.690016 1150 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:43015 every 8 connection(s)
I20260619 16:43:11.691146 1023 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887385903558-388-0/minicluster-data/ts-0/data/info.pb
I20260619 16:43:11.699975 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu as pid 1023
I20260619 16:43:11.700157 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887385903558-388-0/minicluster-data/ts-0/wal/instance
I20260619 16:43:11.700798 388 external_mini_cluster.cc:1428] Setting key de8a85b3968aa7392d9f85b6e29b7931
Jun 19 16:43:11 dist-test-slave-wvz7 krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887391, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 16:43:11.710708 1153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:11.693560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:59125 (local address 127.0.97.62:43151)
0619 16:43:11.694045 (+ 485us) server_negotiation.cc:206] Beginning negotiation
0619 16:43:11.694051 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:43:11.694995 (+ 944us) server_negotiation.cc:407] Connection header received
0619 16:43:11.696675 (+ 1680us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:43:11.696679 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:43:11.696746 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:43:11.696837 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:43:11.698262 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:11.698961 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:11.700548 (+ 1587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:11.701121 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:11.701294 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:43:11.704830 (+ 3536us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:43:11.704837 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:43:11.704841 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:43:11.704886 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:43:11.707610 (+ 2724us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:43:11.708491 (+ 881us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:43:11.708497 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:43:11.708518 (+ 21us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:43:11.708670 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:43:11.709083 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:43:11.709087 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:43:11.709090 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:43:11.709401 (+ 311us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:43:11.709532 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0619 16:43:11.710188 (+ 656us) server_negotiation.cc:299] Negotiation successful
0619 16:43:11.710390 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":359,"thread_start_us":226,"threads_started":1}
I20260619 16:43:11.711549 1154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:11.694104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43151 (local address 127.0.97.1:59125)
0619 16:43:11.694829 (+ 725us) negotiation.cc:107] Waiting for socket to connect
0619 16:43:11.694870 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0619 16:43:11.696447 (+ 1577us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:43:11.697019 (+ 572us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:43:11.697027 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:43:11.697519 (+ 492us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:43:11.698117 (+ 598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:43:11.698133 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:43:11.699099 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:43:11.699103 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:43:11.700420 (+ 1317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:43:11.700431 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:43:11.701356 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:43:11.701362 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:43:11.701634 (+ 272us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:43:11.702355 (+ 721us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:43:11.702376 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:43:11.704625 (+ 2249us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:43:11.707764 (+ 3139us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:43:11.707773 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:43:11.707789 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:43:11.708355 (+ 566us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:43:11.708777 (+ 422us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:43:11.708780 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:43:11.708783 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:43:11.708984 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:43:11.709500 (+ 516us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:43:11.709505 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:43:11.709830 (+ 325us) client_negotiation.cc:769] Sending connection context
0619 16:43:11.710077 (+ 247us) client_negotiation.cc:240] Negotiation successful
0619 16:43:11.710430 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":475,"thread_start_us":273,"threads_started":1}
I20260619 16:43:11.713016 1151 heartbeater.cc:347] Connected to a master server at 127.0.97.62:43151
I20260619 16:43:11.713346 1151 heartbeater.cc:464] Registering TS with master...
I20260619 16:43:11.714205 1151 heartbeater.cc:511] Master 127.0.97.62:43151 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260619 16:43:11.716362 962 ts_manager.cc:194] Registered new tserver with Master: 35efb5d26f4f4307b2f5723abb6749ee (127.0.97.1:43015)
I20260619 16:43:11.717825 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:59125
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887385903558-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887385903558-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260619 16:43:11.737648 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu
/tmp/dist-test-taskEs2EH8/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887385903558-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887385903558-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887385903558-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887385903558-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43151
--keytab_file=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887385903558-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37577
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887385903558-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887385903558-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887385903558-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:43:12.044412 1157 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:43:12.044730 1157 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:43:12.044914 1157 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:43:12.053515 1157 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:43:12.053640 1157 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:43:12.053849 1157 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260619 16:43:12.063606 1157 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37577
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887385903558-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887385903558-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887385903558-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887385903558-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43151
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887385903558-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:43:12.064893 1157 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:43:12.066254 1157 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:43:12.078729 1162 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:43:12.078644 1163 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:43:12.078686 1165 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:43:12.079512 1157 server_base.cc:1034] running on GCE node
I20260619 16:43:12.080119 1157 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:43:12.081038 1157 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:43:12.082228 1157 hybrid_clock.cc:648] HybridClock initialized: now 1781887392082180 us; error 42 us; skew 500 ppm
I20260619 16:43:12.082520 1157 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:43:12 dist-test-slave-wvz7 krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887392, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:43:12.085796 1157 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260619 16:43:12.087213 1157 webserver.cc:460] Webserver started at http://127.0.97.2:35305/ using document root <none> and password file <none>
I20260619 16:43:12.087903 1157 fs_manager.cc:359] Metadata directory not provided
I20260619 16:43:12.088023 1157 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:43:12.088318 1157 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:43:12.091245 1157 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887385903558-388-0/minicluster-data/ts-1/data/instance:
uuid: "d3c970a1f8014401a487a8a72a07a551"
format_stamp: "Formatted at 2026-06-19 16:43:12 on dist-test-slave-wvz7"
server_key: "5cf6aadcbf93fd3e1db3b959bb241865"
server_key_iv: "7a5d6b6ef99c19bb1e15287c797b0615"
server_key_version: "encryptionkey@0"
I20260619 16:43:12.092006 1157 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887385903558-388-0/minicluster-data/ts-1/wal/instance:
uuid: "d3c970a1f8014401a487a8a72a07a551"
format_stamp: "Formatted at 2026-06-19 16:43:12 on dist-test-slave-wvz7"
server_key: "5cf6aadcbf93fd3e1db3b959bb241865"
server_key_iv: "7a5d6b6ef99c19bb1e15287c797b0615"
server_key_version: "encryptionkey@0"
I20260619 16:43:12.096900 1157 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260619 16:43:12.100221 1172 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:43:12.101562 1157 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 16:43:12.101707 1157 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887385903558-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887385903558-388-0/minicluster-data/ts-1/wal
uuid: "d3c970a1f8014401a487a8a72a07a551"
format_stamp: "Formatted at 2026-06-19 16:43:12 on dist-test-slave-wvz7"
server_key: "5cf6aadcbf93fd3e1db3b959bb241865"
server_key_iv: "7a5d6b6ef99c19bb1e15287c797b0615"
server_key_version: "encryptionkey@0"
I20260619 16:43:12.101936 1157 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887385903558-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887385903558-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887385903558-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:43:12.113993 1157 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:43:12.117604 1157 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:43:12.117888 1157 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:43:12.118757 1157 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:43:12.120042 1157 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:43:12.120119 1157 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:43:12.120223 1157 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:43:12.120276 1157 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:43:12.148422 1157 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:36197
I20260619 16:43:12.148434 1284 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:36197 every 8 connection(s)
I20260619 16:43:12.149947 1157 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887385903558-388-0/minicluster-data/ts-1/data/info.pb
I20260619 16:43:12.151749 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu as pid 1157
I20260619 16:43:12.151937 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887385903558-388-0/minicluster-data/ts-1/wal/instance
I20260619 16:43:12.152438 388 external_mini_cluster.cc:1428] Setting key 76dc80f695b9d71437999373910e324f
Jun 19 16:43:12 dist-test-slave-wvz7 krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887392, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
I20260619 16:43:12.171737 1153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:12.152567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:58331 (local address 127.0.97.62:43151)
0619 16:43:12.152763 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0619 16:43:12.152770 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:43:12.154209 (+ 1439us) server_negotiation.cc:407] Connection header received
0619 16:43:12.156188 (+ 1979us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:43:12.156194 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:43:12.156297 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:43:12.156392 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:43:12.157959 (+ 1567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:12.158703 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:12.160827 (+ 2124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:12.161492 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:12.161615 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:43:12.166016 (+ 4401us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:43:12.166022 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:43:12.166025 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:43:12.166060 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:43:12.168486 (+ 2426us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:43:12.169381 (+ 895us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:43:12.169386 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:43:12.169388 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:43:12.169537 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:43:12.170030 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:43:12.170034 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:43:12.170037 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:43:12.170329 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:43:12.170465 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0619 16:43:12.171246 (+ 781us) server_negotiation.cc:299] Negotiation successful
0619 16:43:12.171470 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260619 16:43:12.172823 1287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:12.153157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43151 (local address 127.0.97.2:58331)
0619 16:43:12.154044 (+ 887us) negotiation.cc:107] Waiting for socket to connect
0619 16:43:12.154091 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0619 16:43:12.155911 (+ 1820us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:43:12.156642 (+ 731us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:43:12.156650 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:43:12.157150 (+ 500us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:43:12.157776 (+ 626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:43:12.157803 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:43:12.158877 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:43:12.158881 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:43:12.160669 (+ 1788us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:43:12.160685 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:43:12.161728 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:43:12.161764 (+ 36us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:43:12.162067 (+ 303us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:43:12.162832 (+ 765us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:43:12.162864 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:43:12.165707 (+ 2843us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:43:12.168650 (+ 2943us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:43:12.168660 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:43:12.168679 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:43:12.169228 (+ 549us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:43:12.169641 (+ 413us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:43:12.169645 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:43:12.169647 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:43:12.169891 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:43:12.170461 (+ 570us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:43:12.170469 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:43:12.170829 (+ 360us) client_negotiation.cc:769] Sending connection context
0619 16:43:12.171086 (+ 257us) client_negotiation.cc:240] Negotiation successful
0619 16:43:12.171645 (+ 559us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":603,"thread_start_us":346,"threads_started":1}
I20260619 16:43:12.174247 1285 heartbeater.cc:347] Connected to a master server at 127.0.97.62:43151
I20260619 16:43:12.174542 1285 heartbeater.cc:464] Registering TS with master...
I20260619 16:43:12.175318 1285 heartbeater.cc:511] Master 127.0.97.62:43151 requested a full tablet report, sending...
I20260619 16:43:12.176973 962 ts_manager.cc:194] Registered new tserver with Master: d3c970a1f8014401a487a8a72a07a551 (127.0.97.2:36197)
I20260619 16:43:12.178052 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:58331
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887385903558-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887385903558-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260619 16:43:12.192711 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu
/tmp/dist-test-taskEs2EH8/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887385903558-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887385903558-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887385903558-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887385903558-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43151
--keytab_file=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887385903558-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37577
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887385903558-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887385903558-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887385903558-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:43:12.499614 1290 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:43:12.499892 1290 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:43:12.499985 1290 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:43:12.507174 1290 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:43:12.507298 1290 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:43:12.507426 1290 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260619 16:43:12.517134 1290 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37577
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887385903558-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887385903558-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887385903558-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887385903558-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43151
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887385903558-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:43:12.518357 1290 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:43:12.519589 1290 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:43:12.533336 1298 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:43:12.533293 1295 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:43:12.533368 1296 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:43:12.534086 1290 server_base.cc:1034] running on GCE node
I20260619 16:43:12.534678 1290 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:43:12.535601 1290 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:43:12.536855 1290 hybrid_clock.cc:648] HybridClock initialized: now 1781887392536777 us; error 64 us; skew 500 ppm
I20260619 16:43:12.537175 1290 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:43:12 dist-test-slave-wvz7 krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887392, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:43:12.540105 1290 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260619 16:43:12.541499 1290 webserver.cc:460] Webserver started at http://127.0.97.3:38235/ using document root <none> and password file <none>
I20260619 16:43:12.542187 1290 fs_manager.cc:359] Metadata directory not provided
I20260619 16:43:12.542311 1290 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:43:12.542588 1290 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:43:12.545399 1290 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887385903558-388-0/minicluster-data/ts-2/data/instance:
uuid: "fd2f035248b3420b9424eeaca338b431"
format_stamp: "Formatted at 2026-06-19 16:43:12 on dist-test-slave-wvz7"
server_key: "5cea3aecfc5ac78307f0bb93cbc05d63"
server_key_iv: "6aec453d9d94f78d70be3def08516000"
server_key_version: "encryptionkey@0"
I20260619 16:43:12.546475 1290 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887385903558-388-0/minicluster-data/ts-2/wal/instance:
uuid: "fd2f035248b3420b9424eeaca338b431"
format_stamp: "Formatted at 2026-06-19 16:43:12 on dist-test-slave-wvz7"
server_key: "5cea3aecfc5ac78307f0bb93cbc05d63"
server_key_iv: "6aec453d9d94f78d70be3def08516000"
server_key_version: "encryptionkey@0"
I20260619 16:43:12.551285 1290 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260619 16:43:12.554455 1305 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:43:12.555692 1290 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 16:43:12.555830 1290 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887385903558-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887385903558-388-0/minicluster-data/ts-2/wal
uuid: "fd2f035248b3420b9424eeaca338b431"
format_stamp: "Formatted at 2026-06-19 16:43:12 on dist-test-slave-wvz7"
server_key: "5cea3aecfc5ac78307f0bb93cbc05d63"
server_key_iv: "6aec453d9d94f78d70be3def08516000"
server_key_version: "encryptionkey@0"
I20260619 16:43:12.556028 1290 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887385903558-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887385903558-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887385903558-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:43:12.567538 1290 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:43:12.571146 1290 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:43:12.571394 1290 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:43:12.572261 1290 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:43:12.573521 1290 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:43:12.573592 1290 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:43:12.573707 1290 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:43:12.573776 1290 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260619 16:43:12.603808 1290 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:45483
I20260619 16:43:12.603858 1417 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:45483 every 8 connection(s)
I20260619 16:43:12.605053 1290 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887385903558-388-0/minicluster-data/ts-2/data/info.pb
I20260619 16:43:12.608098 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu as pid 1290
I20260619 16:43:12.608287 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887385903558-388-0/minicluster-data/ts-2/wal/instance
I20260619 16:43:12.608845 388 external_mini_cluster.cc:1428] Setting key 76c010c6d670eda92dda91b9e1ea7749
Jun 19 16:43:12 dist-test-slave-wvz7 krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887392, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 16:43:12.625166 1153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:12.607536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:42081 (local address 127.0.97.62:43151)
0619 16:43:12.607686 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0619 16:43:12.607691 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:43:12.609588 (+ 1897us) server_negotiation.cc:407] Connection header received
0619 16:43:12.611851 (+ 2263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:43:12.611855 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:43:12.611920 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:43:12.611995 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:43:12.613361 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:12.614128 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:12.615632 (+ 1504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:12.616109 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:12.616227 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:43:12.619472 (+ 3245us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:43:12.619477 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:43:12.619480 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:43:12.619519 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:43:12.622002 (+ 2483us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:43:12.622907 (+ 905us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:43:12.622914 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:43:12.622917 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:43:12.623075 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:43:12.623520 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:43:12.623524 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:43:12.623528 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:43:12.623889 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:43:12.624019 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0619 16:43:12.624760 (+ 741us) server_negotiation.cc:299] Negotiation successful
0619 16:43:12.624934 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260619 16:43:12.626183 1420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:12.608141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43151 (local address 127.0.97.3:42081)
0619 16:43:12.609399 (+ 1258us) negotiation.cc:107] Waiting for socket to connect
0619 16:43:12.609456 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0619 16:43:12.611572 (+ 2116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:43:12.612170 (+ 598us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:43:12.612178 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:43:12.612666 (+ 488us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:43:12.613226 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:43:12.613242 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:43:12.614264 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:43:12.614267 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:43:12.615505 (+ 1238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:43:12.615521 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:43:12.616281 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:43:12.616285 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:43:12.616475 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:43:12.617084 (+ 609us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:43:12.617105 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:43:12.619253 (+ 2148us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:43:12.622152 (+ 2899us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:43:12.622160 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:43:12.622176 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:43:12.622711 (+ 535us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:43:12.623185 (+ 474us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:43:12.623188 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:43:12.623190 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:43:12.623415 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:43:12.624025 (+ 610us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:43:12.624033 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:43:12.624434 (+ 401us) client_negotiation.cc:769] Sending connection context
0619 16:43:12.624675 (+ 241us) client_negotiation.cc:240] Negotiation successful
0619 16:43:12.625084 (+ 409us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1001,"thread_start_us":298,"threads_started":1}
I20260619 16:43:12.627408 1418 heartbeater.cc:347] Connected to a master server at 127.0.97.62:43151
I20260619 16:43:12.627660 1418 heartbeater.cc:464] Registering TS with master...
I20260619 16:43:12.628304 1418 heartbeater.cc:511] Master 127.0.97.62:43151 requested a full tablet report, sending...
I20260619 16:43:12.629637 962 ts_manager.cc:194] Registered new tserver with Master: fd2f035248b3420b9424eeaca338b431 (127.0.97.3:45483)
I20260619 16:43:12.630501 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:42081
I20260619 16:43:12.636494 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 16:43:12.654904 1153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:12.642944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39118 (local address 127.0.97.62:43151)
0619 16:43:12.643117 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0619 16:43:12.643121 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:43:12.643645 (+ 524us) server_negotiation.cc:407] Connection header received
0619 16:43:12.643796 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:43:12.643800 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:43:12.643857 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:43:12.643939 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:43:12.645330 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:12.646189 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:12.647499 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:12.647871 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:12.647961 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:43:12.650290 (+ 2329us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:43:12.650295 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:43:12.650298 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:43:12.650331 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:43:12.652669 (+ 2338us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:43:12.653253 (+ 584us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:43:12.653256 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:43:12.653258 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:43:12.653370 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:43:12.653633 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:43:12.653635 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:43:12.653637 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:43:12.653949 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:43:12.654121 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0619 16:43:12.654450 (+ 329us) server_negotiation.cc:299] Negotiation successful
0619 16:43:12.654621 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260619 16:43:12.660846 962 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39118:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 16:43:12.667676 962 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 16:43:12.701399 1432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:12.679024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55248 (local address 127.0.97.1:43015)
0619 16:43:12.690492 (+ 11468us) server_negotiation.cc:206] Beginning negotiation
0619 16:43:12.690504 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 16:43:12.691394 (+ 890us) server_negotiation.cc:407] Connection header received
0619 16:43:12.691819 (+ 425us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:43:12.691829 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:43:12.692105 (+ 276us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:43:12.692269 (+ 164us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:43:12.693369 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:12.695226 (+ 1857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:12.698152 (+ 2926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:12.699736 (+ 1584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:12.699852 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:43:12.700087 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0619 16:43:12.700865 (+ 778us) server_negotiation.cc:299] Negotiation successful
0619 16:43:12.701040 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11272,"thread_start_us":11490,"threads_started":1}
I20260619 16:43:12.702346 1430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:12.678957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:43015 (local address 127.0.0.1:55248)
0619 16:43:12.691232 (+ 12275us) negotiation.cc:107] Waiting for socket to connect
0619 16:43:12.691269 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0619 16:43:12.691556 (+ 287us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:43:12.692656 (+ 1100us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:43:12.692664 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:43:12.692700 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:43:12.693179 (+ 479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:43:12.693193 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:43:12.695440 (+ 2247us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:43:12.695446 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:43:12.697972 (+ 2526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:43:12.697988 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:43:12.700266 (+ 2278us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:43:12.700272 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:43:12.700553 (+ 281us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:43:12.700618 (+ 65us) client_negotiation.cc:769] Sending connection context
0619 16:43:12.701792 (+ 1174us) client_negotiation.cc:240] Negotiation successful
0619 16:43:12.702007 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12038,"thread_start_us":21499,"threads_started":1}
I20260619 16:43:12.706545 1086 tablet_service.cc:1467] Processing CreateTablet for tablet cbaf3007b0134212af6a6c15b13d75c1 (DEFAULT_TABLE table=test-table [id=b3d0bb61b28940d1b4fbc9b75f065812]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:43:12.708076 1086 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cbaf3007b0134212af6a6c15b13d75c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:43:12.709293 1433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:12.679358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59996 (local address 127.0.97.3:45483)
0619 16:43:12.700433 (+ 21075us) server_negotiation.cc:206] Beginning negotiation
0619 16:43:12.700442 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 16:43:12.701065 (+ 623us) server_negotiation.cc:407] Connection header received
0619 16:43:12.701176 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:43:12.701184 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:43:12.701474 (+ 290us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:43:12.702341 (+ 867us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:43:12.702601 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:12.704089 (+ 1488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:12.706455 (+ 2366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:12.707797 (+ 1342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:12.707949 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:43:12.708157 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0619 16:43:12.708832 (+ 675us) server_negotiation.cc:299] Negotiation successful
0619 16:43:12.708997 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20895,"thread_start_us":20727,"threads_started":1}
I20260619 16:43:12.709406 1431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:12.679021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:45483 (local address 127.0.0.1:59996)
0619 16:43:12.700725 (+ 21704us) negotiation.cc:107] Waiting for socket to connect
0619 16:43:12.700749 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0619 16:43:12.700924 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:43:12.701694 (+ 770us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:43:12.701699 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:43:12.701726 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:43:12.702150 (+ 424us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:43:12.702162 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:43:12.704227 (+ 2065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:43:12.704233 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:43:12.706278 (+ 2045us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:43:12.706292 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:43:12.707974 (+ 1682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:43:12.707980 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:43:12.708134 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:43:12.708168 (+ 34us) client_negotiation.cc:769] Sending connection context
0619 16:43:12.708252 (+ 84us) client_negotiation.cc:240] Negotiation successful
0619 16:43:12.708390 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":21572,"mutex_wait_us":1,"thread_start_us":11845,"threads_started":1}
I20260619 16:43:12.709944 1435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:12.698098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:36197 (local address 127.0.0.1:57064)
0619 16:43:12.701083 (+ 2985us) negotiation.cc:107] Waiting for socket to connect
0619 16:43:12.701098 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 16:43:12.701193 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:43:12.702523 (+ 1330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:43:12.702528 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:43:12.702550 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:43:12.703401 (+ 851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:43:12.703415 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:43:12.705059 (+ 1644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:43:12.705064 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:43:12.707263 (+ 2199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:43:12.707286 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:43:12.709208 (+ 1922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:43:12.709213 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:43:12.709365 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:43:12.709396 (+ 31us) client_negotiation.cc:769] Sending connection context
0619 16:43:12.709479 (+ 83us) client_negotiation.cc:240] Negotiation successful
0619 16:43:12.709600 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2856,"thread_start_us":2716,"threads_started":1}
I20260619 16:43:12.710273 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:12.692128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57064 (local address 127.0.97.2:36197)
0619 16:43:12.701672 (+ 9544us) server_negotiation.cc:206] Beginning negotiation
0619 16:43:12.701681 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 16:43:12.701708 (+ 27us) server_negotiation.cc:407] Connection header received
0619 16:43:12.701834 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:43:12.701841 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:43:12.702088 (+ 247us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:43:12.702283 (+ 195us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:43:12.703572 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:12.704889 (+ 1317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:12.707407 (+ 2518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:12.708804 (+ 1397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:12.708921 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:43:12.709088 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0619 16:43:12.709817 (+ 729us) server_negotiation.cc:299] Negotiation successful
0619 16:43:12.709975 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9386,"thread_start_us":9254,"threads_started":1}
I20260619 16:43:12.713052 1220 tablet_service.cc:1467] Processing CreateTablet for tablet cbaf3007b0134212af6a6c15b13d75c1 (DEFAULT_TABLE table=test-table [id=b3d0bb61b28940d1b4fbc9b75f065812]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:43:12.713047 1353 tablet_service.cc:1467] Processing CreateTablet for tablet cbaf3007b0134212af6a6c15b13d75c1 (DEFAULT_TABLE table=test-table [id=b3d0bb61b28940d1b4fbc9b75f065812]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:43:12.714322 1220 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cbaf3007b0134212af6a6c15b13d75c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:43:12.714708 1353 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cbaf3007b0134212af6a6c15b13d75c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:43:12.721076 1151 heartbeater.cc:503] Master 127.0.97.62:43151 was elected leader, sending a full tablet report...
I20260619 16:43:12.724819 1436 tablet_bootstrap.cc:492] T cbaf3007b0134212af6a6c15b13d75c1 P 35efb5d26f4f4307b2f5723abb6749ee: Bootstrap starting.
I20260619 16:43:12.726630 1437 tablet_bootstrap.cc:492] T cbaf3007b0134212af6a6c15b13d75c1 P d3c970a1f8014401a487a8a72a07a551: Bootstrap starting.
I20260619 16:43:12.728478 1436 tablet_bootstrap.cc:654] T cbaf3007b0134212af6a6c15b13d75c1 P 35efb5d26f4f4307b2f5723abb6749ee: Neither blocks nor log segments found. Creating new log.
I20260619 16:43:12.729442 1436 log.cc:824] T cbaf3007b0134212af6a6c15b13d75c1 P 35efb5d26f4f4307b2f5723abb6749ee: Log is configured to *not* fsync() on all Append() calls
I20260619 16:43:12.730320 1437 tablet_bootstrap.cc:654] T cbaf3007b0134212af6a6c15b13d75c1 P d3c970a1f8014401a487a8a72a07a551: Neither blocks nor log segments found. Creating new log.
I20260619 16:43:12.731700 1437 log.cc:824] T cbaf3007b0134212af6a6c15b13d75c1 P d3c970a1f8014401a487a8a72a07a551: Log is configured to *not* fsync() on all Append() calls
I20260619 16:43:12.732116 1438 tablet_bootstrap.cc:492] T cbaf3007b0134212af6a6c15b13d75c1 P fd2f035248b3420b9424eeaca338b431: Bootstrap starting.
I20260619 16:43:12.733238 1436 tablet_bootstrap.cc:492] T cbaf3007b0134212af6a6c15b13d75c1 P 35efb5d26f4f4307b2f5723abb6749ee: No bootstrap required, opened a new log
I20260619 16:43:12.733513 1436 ts_tablet_manager.cc:1397] T cbaf3007b0134212af6a6c15b13d75c1 P 35efb5d26f4f4307b2f5723abb6749ee: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260619 16:43:12.735661 1438 tablet_bootstrap.cc:654] T cbaf3007b0134212af6a6c15b13d75c1 P fd2f035248b3420b9424eeaca338b431: Neither blocks nor log segments found. Creating new log.
I20260619 16:43:12.736814 1438 log.cc:824] T cbaf3007b0134212af6a6c15b13d75c1 P fd2f035248b3420b9424eeaca338b431: Log is configured to *not* fsync() on all Append() calls
I20260619 16:43:12.738193 1437 tablet_bootstrap.cc:492] T cbaf3007b0134212af6a6c15b13d75c1 P d3c970a1f8014401a487a8a72a07a551: No bootstrap required, opened a new log
I20260619 16:43:12.738461 1437 ts_tablet_manager.cc:1397] T cbaf3007b0134212af6a6c15b13d75c1 P d3c970a1f8014401a487a8a72a07a551: Time spent bootstrapping tablet: real 0.012s user 0.007s sys 0.004s
I20260619 16:43:12.740643 1438 tablet_bootstrap.cc:492] T cbaf3007b0134212af6a6c15b13d75c1 P fd2f035248b3420b9424eeaca338b431: No bootstrap required, opened a new log
I20260619 16:43:12.740890 1438 ts_tablet_manager.cc:1397] T cbaf3007b0134212af6a6c15b13d75c1 P fd2f035248b3420b9424eeaca338b431: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.007s
I20260619 16:43:12.742987 1436 raft_consensus.cc:348] T cbaf3007b0134212af6a6c15b13d75c1 P 35efb5d26f4f4307b2f5723abb6749ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35efb5d26f4f4307b2f5723abb6749ee" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43015 } } peers { permanent_uuid: "fd2f035248b3420b9424eeaca338b431" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45483 } } peers { permanent_uuid: "d3c970a1f8014401a487a8a72a07a551" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36197 } }
I20260619 16:43:12.743402 1436 raft_consensus.cc:374] T cbaf3007b0134212af6a6c15b13d75c1 P 35efb5d26f4f4307b2f5723abb6749ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:43:12.743530 1436 raft_consensus.cc:729] T cbaf3007b0134212af6a6c15b13d75c1 P 35efb5d26f4f4307b2f5723abb6749ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35efb5d26f4f4307b2f5723abb6749ee, State: Initialized, Role: FOLLOWER
I20260619 16:43:12.744171 1436 consensus_queue.cc:260] T cbaf3007b0134212af6a6c15b13d75c1 P 35efb5d26f4f4307b2f5723abb6749ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35efb5d26f4f4307b2f5723abb6749ee" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43015 } } peers { permanent_uuid: "fd2f035248b3420b9424eeaca338b431" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45483 } } peers { permanent_uuid: "d3c970a1f8014401a487a8a72a07a551" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36197 } }
I20260619 16:43:12.746008 1436 ts_tablet_manager.cc:1428] T cbaf3007b0134212af6a6c15b13d75c1 P 35efb5d26f4f4307b2f5723abb6749ee: Time spent starting tablet: real 0.012s user 0.014s sys 0.000s
I20260619 16:43:12.746855 1437 raft_consensus.cc:348] T cbaf3007b0134212af6a6c15b13d75c1 P d3c970a1f8014401a487a8a72a07a551 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35efb5d26f4f4307b2f5723abb6749ee" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43015 } } peers { permanent_uuid: "fd2f035248b3420b9424eeaca338b431" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45483 } } peers { permanent_uuid: "d3c970a1f8014401a487a8a72a07a551" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36197 } }
I20260619 16:43:12.747257 1437 raft_consensus.cc:374] T cbaf3007b0134212af6a6c15b13d75c1 P d3c970a1f8014401a487a8a72a07a551 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:43:12.747344 1437 raft_consensus.cc:729] T cbaf3007b0134212af6a6c15b13d75c1 P d3c970a1f8014401a487a8a72a07a551 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3c970a1f8014401a487a8a72a07a551, State: Initialized, Role: FOLLOWER
I20260619 16:43:12.747931 1437 consensus_queue.cc:260] T cbaf3007b0134212af6a6c15b13d75c1 P d3c970a1f8014401a487a8a72a07a551 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35efb5d26f4f4307b2f5723abb6749ee" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43015 } } peers { permanent_uuid: "fd2f035248b3420b9424eeaca338b431" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45483 } } peers { permanent_uuid: "d3c970a1f8014401a487a8a72a07a551" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36197 } }
I20260619 16:43:12.749001 1285 heartbeater.cc:503] Master 127.0.97.62:43151 was elected leader, sending a full tablet report...
I20260619 16:43:12.751140 1437 ts_tablet_manager.cc:1428] T cbaf3007b0134212af6a6c15b13d75c1 P d3c970a1f8014401a487a8a72a07a551: Time spent starting tablet: real 0.013s user 0.010s sys 0.000s
I20260619 16:43:12.751247 1438 raft_consensus.cc:348] T cbaf3007b0134212af6a6c15b13d75c1 P fd2f035248b3420b9424eeaca338b431 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35efb5d26f4f4307b2f5723abb6749ee" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43015 } } peers { permanent_uuid: "fd2f035248b3420b9424eeaca338b431" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45483 } } peers { permanent_uuid: "d3c970a1f8014401a487a8a72a07a551" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36197 } }
I20260619 16:43:12.751659 1438 raft_consensus.cc:374] T cbaf3007b0134212af6a6c15b13d75c1 P fd2f035248b3420b9424eeaca338b431 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:43:12.751754 1438 raft_consensus.cc:729] T cbaf3007b0134212af6a6c15b13d75c1 P fd2f035248b3420b9424eeaca338b431 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd2f035248b3420b9424eeaca338b431, State: Initialized, Role: FOLLOWER
I20260619 16:43:12.752329 1438 consensus_queue.cc:260] T cbaf3007b0134212af6a6c15b13d75c1 P fd2f035248b3420b9424eeaca338b431 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35efb5d26f4f4307b2f5723abb6749ee" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43015 } } peers { permanent_uuid: "fd2f035248b3420b9424eeaca338b431" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45483 } } peers { permanent_uuid: "d3c970a1f8014401a487a8a72a07a551" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36197 } }
I20260619 16:43:12.753316 1418 heartbeater.cc:503] Master 127.0.97.62:43151 was elected leader, sending a full tablet report...
I20260619 16:43:12.753852 1438 ts_tablet_manager.cc:1428] T cbaf3007b0134212af6a6c15b13d75c1 P fd2f035248b3420b9424eeaca338b431: Time spent starting tablet: real 0.013s user 0.013s sys 0.000s
W20260619 16:43:12.856537 1419 tablet.cc:2369] T cbaf3007b0134212af6a6c15b13d75c1 P fd2f035248b3420b9424eeaca338b431: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 16:43:12.867090 1444 raft_consensus.cc:482] T cbaf3007b0134212af6a6c15b13d75c1 P fd2f035248b3420b9424eeaca338b431 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 16:43:12.867365 1444 raft_consensus.cc:504] T cbaf3007b0134212af6a6c15b13d75c1 P fd2f035248b3420b9424eeaca338b431 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35efb5d26f4f4307b2f5723abb6749ee" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43015 } } peers { permanent_uuid: "fd2f035248b3420b9424eeaca338b431" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45483 } } peers { permanent_uuid: "d3c970a1f8014401a487a8a72a07a551" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36197 } }
I20260619 16:43:12.868990 1444 leader_election.cc:288] T cbaf3007b0134212af6a6c15b13d75c1 P fd2f035248b3420b9424eeaca338b431 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 35efb5d26f4f4307b2f5723abb6749ee (127.0.97.1:43015), d3c970a1f8014401a487a8a72a07a551 (127.0.97.2:36197)
I20260619 16:43:12.874045 1420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:12.869328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:43015 (local address 127.0.97.3:53377)
0619 16:43:12.869505 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0619 16:43:12.869526 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 16:43:12.869654 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:43:12.869988 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:43:12.869997 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:43:12.870023 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:43:12.870207 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:43:12.870219 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:43:12.871155 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:43:12.871160 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:43:12.872357 (+ 1197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:43:12.872367 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:43:12.873369 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:43:12.873375 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:43:12.873535 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:43:12.873557 (+ 22us) client_negotiation.cc:769] Sending connection context
0619 16:43:12.873635 (+ 78us) client_negotiation.cc:240] Negotiation successful
0619 16:43:12.873784 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42}
I20260619 16:43:12.874051 1432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:12.869333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:53377 (local address 127.0.97.1:43015)
0619 16:43:12.869511 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0619 16:43:12.869517 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:43:12.869600 (+ 83us) server_negotiation.cc:407] Connection header received
0619 16:43:12.869807 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:43:12.869813 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:43:12.869872 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:43:12.869957 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:43:12.870336 (+ 379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:12.871037 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:12.872501 (+ 1464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:12.873228 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:12.873343 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:43:12.873449 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0619 16:43:12.873665 (+ 216us) server_negotiation.cc:299] Negotiation successful
0619 16:43:12.873823 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260619 16:43:12.874943 1106 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cbaf3007b0134212af6a6c15b13d75c1" candidate_uuid: "fd2f035248b3420b9424eeaca338b431" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "35efb5d26f4f4307b2f5723abb6749ee" is_pre_election: true
I20260619 16:43:12.875415 1106 raft_consensus.cc:2454] T cbaf3007b0134212af6a6c15b13d75c1 P 35efb5d26f4f4307b2f5723abb6749ee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fd2f035248b3420b9424eeaca338b431 in term 0.
I20260619 16:43:12.875571 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:12.869523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:45139 (local address 127.0.97.2:36197)
0619 16:43:12.869693 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0619 16:43:12.869699 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:43:12.870492 (+ 793us) server_negotiation.cc:407] Connection header received
0619 16:43:12.870634 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:43:12.870638 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:43:12.870691 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:43:12.870761 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:43:12.871218 (+ 457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:12.871871 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:12.873411 (+ 1540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:12.874749 (+ 1338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:12.874885 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:43:12.874971 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0619 16:43:12.875203 (+ 232us) server_negotiation.cc:299] Negotiation successful
0619 16:43:12.875331 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260619 16:43:12.875638 1445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:12.869333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:36197 (local address 127.0.97.3:45139)
0619 16:43:12.870395 (+ 1062us) negotiation.cc:107] Waiting for socket to connect
0619 16:43:12.870409 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 16:43:12.870517 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:43:12.870791 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:43:12.870795 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:43:12.870818 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:43:12.871082 (+ 264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:43:12.871090 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:43:12.872006 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:43:12.872010 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:43:12.873269 (+ 1259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:43:12.873280 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:43:12.874922 (+ 1642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:43:12.874928 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:43:12.875065 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:43:12.875088 (+ 23us) client_negotiation.cc:769] Sending connection context
0619 16:43:12.875163 (+ 75us) client_negotiation.cc:240] Negotiation successful
0619 16:43:12.875278 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":973,"mutex_wait_us":461,"thread_start_us":292,"threads_started":1}
I20260619 16:43:12.876087 1308 leader_election.cc:302] T cbaf3007b0134212af6a6c15b13d75c1 P fd2f035248b3420b9424eeaca338b431 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 35efb5d26f4f4307b2f5723abb6749ee, fd2f035248b3420b9424eeaca338b431; no voters:
I20260619 16:43:12.876467 1444 raft_consensus.cc:2781] T cbaf3007b0134212af6a6c15b13d75c1 P fd2f035248b3420b9424eeaca338b431 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 16:43:12.876587 1444 raft_consensus.cc:482] T cbaf3007b0134212af6a6c15b13d75c1 P fd2f035248b3420b9424eeaca338b431 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 16:43:12.876674 1444 raft_consensus.cc:3037] T cbaf3007b0134212af6a6c15b13d75c1 P fd2f035248b3420b9424eeaca338b431 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:43:12.876555 1240 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cbaf3007b0134212af6a6c15b13d75c1" candidate_uuid: "fd2f035248b3420b9424eeaca338b431" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3c970a1f8014401a487a8a72a07a551" is_pre_election: true
I20260619 16:43:12.877004 1240 raft_consensus.cc:2454] T cbaf3007b0134212af6a6c15b13d75c1 P d3c970a1f8014401a487a8a72a07a551 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fd2f035248b3420b9424eeaca338b431 in term 0.
I20260619 16:43:12.878888 1444 raft_consensus.cc:504] T cbaf3007b0134212af6a6c15b13d75c1 P fd2f035248b3420b9424eeaca338b431 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35efb5d26f4f4307b2f5723abb6749ee" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43015 } } peers { permanent_uuid: "fd2f035248b3420b9424eeaca338b431" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45483 } } peers { permanent_uuid: "d3c970a1f8014401a487a8a72a07a551" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36197 } }
I20260619 16:43:12.879419 1444 leader_election.cc:288] T cbaf3007b0134212af6a6c15b13d75c1 P fd2f035248b3420b9424eeaca338b431 [CANDIDATE]: Term 1 election: Requested vote from peers 35efb5d26f4f4307b2f5723abb6749ee (127.0.97.1:43015), d3c970a1f8014401a487a8a72a07a551 (127.0.97.2:36197)
I20260619 16:43:12.879798 1106 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cbaf3007b0134212af6a6c15b13d75c1" candidate_uuid: "fd2f035248b3420b9424eeaca338b431" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "35efb5d26f4f4307b2f5723abb6749ee"
I20260619 16:43:12.879850 1240 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cbaf3007b0134212af6a6c15b13d75c1" candidate_uuid: "fd2f035248b3420b9424eeaca338b431" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3c970a1f8014401a487a8a72a07a551"
I20260619 16:43:12.880012 1240 raft_consensus.cc:3037] T cbaf3007b0134212af6a6c15b13d75c1 P d3c970a1f8014401a487a8a72a07a551 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:43:12.880009 1106 raft_consensus.cc:3037] T cbaf3007b0134212af6a6c15b13d75c1 P 35efb5d26f4f4307b2f5723abb6749ee [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:43:12.883035 1106 raft_consensus.cc:2454] T cbaf3007b0134212af6a6c15b13d75c1 P 35efb5d26f4f4307b2f5723abb6749ee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fd2f035248b3420b9424eeaca338b431 in term 1.
I20260619 16:43:12.883034 1240 raft_consensus.cc:2454] T cbaf3007b0134212af6a6c15b13d75c1 P d3c970a1f8014401a487a8a72a07a551 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fd2f035248b3420b9424eeaca338b431 in term 1.
I20260619 16:43:12.883448 1309 leader_election.cc:302] T cbaf3007b0134212af6a6c15b13d75c1 P fd2f035248b3420b9424eeaca338b431 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 35efb5d26f4f4307b2f5723abb6749ee, d3c970a1f8014401a487a8a72a07a551, fd2f035248b3420b9424eeaca338b431; no voters:
I20260619 16:43:12.883809 1444 raft_consensus.cc:2781] T cbaf3007b0134212af6a6c15b13d75c1 P fd2f035248b3420b9424eeaca338b431 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:43:12.884208 1444 raft_consensus.cc:686] T cbaf3007b0134212af6a6c15b13d75c1 P fd2f035248b3420b9424eeaca338b431 [term 1 LEADER]: Becoming Leader. State: Replica: fd2f035248b3420b9424eeaca338b431, State: Running, Role: LEADER
I20260619 16:43:12.884708 1444 consensus_queue.cc:237] T cbaf3007b0134212af6a6c15b13d75c1 P fd2f035248b3420b9424eeaca338b431 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35efb5d26f4f4307b2f5723abb6749ee" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43015 } } peers { permanent_uuid: "fd2f035248b3420b9424eeaca338b431" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45483 } } peers { permanent_uuid: "d3c970a1f8014401a487a8a72a07a551" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36197 } }
I20260619 16:43:12.889051 960 catalog_manager.cc:5495] T cbaf3007b0134212af6a6c15b13d75c1 P fd2f035248b3420b9424eeaca338b431 reported cstate change: term changed from 0 to 1, leader changed from <none> to fd2f035248b3420b9424eeaca338b431 (127.0.97.3). New cstate: current_term: 1 leader_uuid: "fd2f035248b3420b9424eeaca338b431" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35efb5d26f4f4307b2f5723abb6749ee" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43015 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fd2f035248b3420b9424eeaca338b431" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45483 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d3c970a1f8014401a487a8a72a07a551" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36197 } health_report { overall_health: UNKNOWN } } }
W20260619 16:43:12.902004 1286 tablet.cc:2369] T cbaf3007b0134212af6a6c15b13d75c1 P d3c970a1f8014401a487a8a72a07a551: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 16:43:12.943383 1152 tablet.cc:2369] T cbaf3007b0134212af6a6c15b13d75c1 P 35efb5d26f4f4307b2f5723abb6749ee: Can't schedule compaction. Clean time has not been advanced past its initial value.
Jun 19 16:43:12 dist-test-slave-wvz7 krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887390, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260619 16:43:12.953122 1432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:12.939282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55250 (local address 127.0.97.1:43015)
0619 16:43:12.939435 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0619 16:43:12.939440 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:43:12.939889 (+ 449us) server_negotiation.cc:407] Connection header received
0619 16:43:12.940057 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:43:12.940061 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:43:12.940122 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:43:12.940204 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:43:12.941602 (+ 1398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:12.942397 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:12.943807 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:12.944244 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:12.944358 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:43:12.948143 (+ 3785us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:43:12.948157 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:43:12.948172 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:43:12.948209 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:43:12.950715 (+ 2506us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:43:12.951342 (+ 627us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:43:12.951350 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:43:12.951352 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:43:12.951506 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:43:12.951899 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:43:12.951902 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:43:12.951904 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:43:12.952227 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:43:12.952453 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0619 16:43:12.952704 (+ 251us) server_negotiation.cc:299] Negotiation successful
0619 16:43:12.952855 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
Jun 19 16:43:12 dist-test-slave-wvz7 krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887392, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 19 16:43:12 dist-test-slave-wvz7 krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887392, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260619 16:43:12.993673 1432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:12.981137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55252 (local address 127.0.97.1:43015)
0619 16:43:12.981372 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0619 16:43:12.981377 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:43:12.981493 (+ 116us) server_negotiation.cc:407] Connection header received
0619 16:43:12.981634 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:43:12.981638 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:43:12.981699 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:43:12.981810 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:43:12.983014 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:12.983720 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:12.984991 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:12.985387 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:12.985477 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:43:12.988926 (+ 3449us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:43:12.988931 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:43:12.988934 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:43:12.988964 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:43:12.991391 (+ 2427us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:43:12.992003 (+ 612us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:43:12.992009 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:43:12.992013 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:43:12.992141 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:43:12.992525 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:43:12.992528 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:43:12.992530 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:43:12.992765 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:43:12.992939 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0619 16:43:12.993221 (+ 282us) server_negotiation.cc:299] Negotiation successful
0619 16:43:12.993390 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98}
Jun 19 16:43:13 dist-test-slave-wvz7 krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887393, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 19 16:43:13 dist-test-slave-wvz7 krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887393, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260619 16:43:13.033486 1432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:13.020668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55256 (local address 127.0.97.1:43015)
0619 16:43:13.020916 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0619 16:43:13.020922 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:43:13.021210 (+ 288us) server_negotiation.cc:407] Connection header received
0619 16:43:13.021384 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:43:13.021388 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:43:13.021448 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:43:13.021531 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:43:13.022836 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:13.023503 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:13.024676 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:13.025082 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:13.025188 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:43:13.028701 (+ 3513us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:43:13.028706 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:43:13.028709 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:43:13.028743 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:43:13.031125 (+ 2382us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:43:13.031711 (+ 586us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:43:13.031716 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:43:13.031718 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:43:13.031830 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:43:13.032158 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:43:13.032163 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:43:13.032166 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:43:13.032444 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:43:13.032691 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0619 16:43:13.032939 (+ 248us) server_negotiation.cc:299] Negotiation successful
0619 16:43:13.033163 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
W20260619 16:43:13.035178 1066 tablet_service.cc:2908] Rejecting scan request for tablet cbaf3007b0134212af6a6c15b13d75c1: Uninitialized: clean time has not yet been initialized
I20260619 16:43:13.038122 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu pid 1023
I20260619 16:43:13.050631 1432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:13.038946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55264 (local address 127.0.97.1:43015)
0619 16:43:13.039092 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0619 16:43:13.039097 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:43:13.039434 (+ 337us) server_negotiation.cc:407] Connection header received
0619 16:43:13.039586 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:43:13.039590 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:43:13.039657 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:43:13.039728 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:43:13.040921 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:13.041576 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:13.042810 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:13.043258 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:13.043376 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:43:13.045492 (+ 2116us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:43:13.045497 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:43:13.045500 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:43:13.045529 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:43:13.048352 (+ 2823us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:43:13.049018 (+ 666us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:43:13.049021 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:43:13.049023 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:43:13.049124 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:43:13.049484 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:43:13.049487 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:43:13.049489 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:43:13.049782 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:43:13.049977 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0619 16:43:13.050223 (+ 246us) server_negotiation.cc:299] Negotiation successful
0619 16:43:13.050403 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260619 16:43:13.051249 1146 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:43:13.137646 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu with pid 1023
I20260619 16:43:13.151943 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu pid 1157
Jun 19 16:43:13 dist-test-slave-wvz7 krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887393, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260619 16:43:13.166579 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:13.152805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39830 (local address 127.0.97.2:36197)
0619 16:43:13.152992 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0619 16:43:13.152999 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:43:13.153245 (+ 246us) server_negotiation.cc:407] Connection header received
0619 16:43:13.153423 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:43:13.153428 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:43:13.153510 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:43:13.153602 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:43:13.154995 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:13.156167 (+ 1172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:13.157636 (+ 1469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:13.158252 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:13.158359 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:43:13.161943 (+ 3584us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:43:13.161953 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:43:13.161966 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:43:13.161996 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:43:13.164329 (+ 2333us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:43:13.164879 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:43:13.164885 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:43:13.164887 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:43:13.164996 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:43:13.165269 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:43:13.165272 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:43:13.165274 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:43:13.165596 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:43:13.166037 (+ 441us) server_negotiation.cc:1035] Waiting for connection context
0619 16:43:13.166144 (+ 107us) server_negotiation.cc:299] Negotiation successful
0619 16:43:13.166302 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260619 16:43:13.167212 1280 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:43:13.253651 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu with pid 1157
I20260619 16:43:13.266163 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu pid 1290
Jun 19 16:43:13 dist-test-slave-wvz7 krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887393, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260619 16:43:13.280581 1461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:13.266896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39554 (local address 127.0.97.3:45483)
0619 16:43:13.267334 (+ 438us) server_negotiation.cc:206] Beginning negotiation
0619 16:43:13.267339 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:43:13.267353 (+ 14us) server_negotiation.cc:407] Connection header received
0619 16:43:13.267407 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:43:13.267411 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:43:13.267466 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:43:13.267557 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:43:13.268736 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:13.269415 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:13.270676 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:13.271069 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:13.271179 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:43:13.275072 (+ 3893us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:43:13.275086 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:43:13.275101 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:43:13.275143 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:43:13.277870 (+ 2727us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:43:13.278549 (+ 679us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:43:13.278557 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:43:13.278559 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:43:13.278698 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:43:13.279064 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:43:13.279069 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:43:13.279072 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:43:13.279424 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:43:13.279711 (+ 287us) server_negotiation.cc:1035] Waiting for connection context
0619 16:43:13.279974 (+ 263us) server_negotiation.cc:299] Negotiation successful
0619 16:43:13.280187 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":339,"thread_start_us":216,"threads_started":1}
I20260619 16:43:13.281189 1413 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:43:13.366789 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu with pid 1290
I20260619 16:43:13.366906 1445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:13.366378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:36197 (local address 127.0.97.3:34219)
0619 16:43:13.366565 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0619 16:43:13.366693 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.2:36197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260619 16:43:13.378945 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu pid 930
I20260619 16:43:13.380080 992 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:43:13.483367 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu with pid 930
2026-06-19T16:43:13Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (3879 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887385903558-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 16:43:13 dist-test-slave-wvz7 krb5kdc[1466](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 16:43:13 dist-test-slave-wvz7 krb5kdc[1466](info): set up 2 sockets
Jun 19 16:43:13 dist-test-slave-wvz7 krb5kdc[1466](info): commencing operation
krb5kdc: starting...
W20260619 16:43:14.481429 388 mini_kdc.cc:121] Time spent starting KDC: real 0.961s user 0.003s sys 0.011s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887385903558-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887385903558-388-0/krb5kdc/test-user.keytab.
Jun 19 16:43:14 dist-test-slave-wvz7 krb5kdc[1466](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887394, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T16:43:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T16:43:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887385903558-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887385903558-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260619 16:43:14.642401 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu
/tmp/dist-test-taskEs2EH8/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887385903558-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887385903558-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887385903558-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887385903558-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34657
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887385903558-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38449
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:34657
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887385903558-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887385903558-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887385903558-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:43:14.937251 1479 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:43:14.937533 1479 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:43:14.937646 1479 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:43:14.944648 1479 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:43:14.944752 1479 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260619 16:43:14.944798 1479 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:43:14.944880 1479 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:43:14.944933 1479 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:43:14.945013 1479 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:43:14.954515 1479 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38449
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887385903558-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887385903558-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:34657
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887385903558-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34657
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887385903558-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887385903558-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:43:14.955722 1479 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:43:14.956965 1479 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:43:14.967864 1487 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:43:14.967918 1484 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:43:14.967875 1485 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:43:14.968757 1479 server_base.cc:1034] running on GCE node
I20260619 16:43:14.969491 1479 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:43:14.970914 1479 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:43:14.972132 1479 hybrid_clock.cc:648] HybridClock initialized: now 1781887394972061 us; error 47 us; skew 500 ppm
I20260619 16:43:14.972446 1479 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:43:14 dist-test-slave-wvz7 krb5kdc[1466](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887394, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:43:14.977661 1479 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260619 16:43:14.978977 1479 webserver.cc:460] Webserver started at http://127.0.97.62:46553/ using document root <none> and password file <none>
I20260619 16:43:14.979629 1479 fs_manager.cc:359] Metadata directory not provided
I20260619 16:43:14.979722 1479 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:43:14.980008 1479 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:43:14.982813 1479 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887385903558-388-0/minicluster-data/master-0/data/instance:
uuid: "de31151854fc4a318c6bc1f46f6b59bf"
format_stamp: "Formatted at 2026-06-19 16:43:14 on dist-test-slave-wvz7"
server_key: "54421387448a20f826e9179f9d5c92eb"
server_key_iv: "4b0f6b751694f5735d39954a858e9533"
server_key_version: "encryptionkey@0"
I20260619 16:43:14.983538 1479 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887385903558-388-0/minicluster-data/master-0/wal/instance:
uuid: "de31151854fc4a318c6bc1f46f6b59bf"
format_stamp: "Formatted at 2026-06-19 16:43:14 on dist-test-slave-wvz7"
server_key: "54421387448a20f826e9179f9d5c92eb"
server_key_iv: "4b0f6b751694f5735d39954a858e9533"
server_key_version: "encryptionkey@0"
I20260619 16:43:14.988401 1479 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260619 16:43:14.991514 1494 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:43:14.992722 1479 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260619 16:43:14.992869 1479 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887385903558-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887385903558-388-0/minicluster-data/master-0/wal
uuid: "de31151854fc4a318c6bc1f46f6b59bf"
format_stamp: "Formatted at 2026-06-19 16:43:14 on dist-test-slave-wvz7"
server_key: "54421387448a20f826e9179f9d5c92eb"
server_key_iv: "4b0f6b751694f5735d39954a858e9533"
server_key_version: "encryptionkey@0"
I20260619 16:43:14.993043 1479 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887385903558-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887385903558-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887385903558-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:43:15.013643 1479 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:43:15.017481 1479 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:43:15.017714 1479 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:43:15.037562 1479 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:34657
I20260619 16:43:15.037582 1555 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:34657 every 8 connection(s)
I20260619 16:43:15.038916 1479 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887385903558-388-0/minicluster-data/master-0/data/info.pb
I20260619 16:43:15.042548 1556 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:43:15.046587 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu as pid 1479
I20260619 16:43:15.046797 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887385903558-388-0/minicluster-data/master-0/wal/instance
I20260619 16:43:15.047374 388 external_mini_cluster.cc:1428] Setting key 7e6839ad6ea00ad20cc33db5b776b8c1
I20260619 16:43:15.053987 1556 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P de31151854fc4a318c6bc1f46f6b59bf: Bootstrap starting.
I20260619 16:43:15.057400 1556 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P de31151854fc4a318c6bc1f46f6b59bf: Neither blocks nor log segments found. Creating new log.
I20260619 16:43:15.058542 1556 log.cc:824] T 00000000000000000000000000000000 P de31151854fc4a318c6bc1f46f6b59bf: Log is configured to *not* fsync() on all Append() calls
I20260619 16:43:15.061165 1556 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P de31151854fc4a318c6bc1f46f6b59bf: No bootstrap required, opened a new log
Jun 19 16:43:15 dist-test-slave-wvz7 krb5kdc[1466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887394, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 16:43:15.068698 1556 raft_consensus.cc:348] T 00000000000000000000000000000000 P de31151854fc4a318c6bc1f46f6b59bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de31151854fc4a318c6bc1f46f6b59bf" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34657 } }
I20260619 16:43:15.069020 1556 raft_consensus.cc:374] T 00000000000000000000000000000000 P de31151854fc4a318c6bc1f46f6b59bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:43:15.069130 1556 raft_consensus.cc:729] T 00000000000000000000000000000000 P de31151854fc4a318c6bc1f46f6b59bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: de31151854fc4a318c6bc1f46f6b59bf, State: Initialized, Role: FOLLOWER
I20260619 16:43:15.069658 1556 consensus_queue.cc:260] T 00000000000000000000000000000000 P de31151854fc4a318c6bc1f46f6b59bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de31151854fc4a318c6bc1f46f6b59bf" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34657 } }
I20260619 16:43:15.069892 1559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:15.049347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40210 (local address 127.0.97.62:34657)
0619 16:43:15.050147 (+ 800us) server_negotiation.cc:206] Beginning negotiation
0619 16:43:15.050163 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0619 16:43:15.050210 (+ 47us) server_negotiation.cc:407] Connection header received
0619 16:43:15.052560 (+ 2350us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:43:15.052587 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:43:15.053068 (+ 481us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:43:15.053474 (+ 406us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:43:15.054711 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:15.056654 (+ 1943us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:15.057920 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:15.058911 (+ 991us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:15.059107 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:43:15.063029 (+ 3922us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:43:15.063037 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:43:15.063051 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:43:15.063090 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:43:15.065965 (+ 2875us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:43:15.066576 (+ 611us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:43:15.066582 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:43:15.066588 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:43:15.066746 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:43:15.067087 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:43:15.067094 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:43:15.067096 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:43:15.067603 (+ 507us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:43:15.067875 (+ 272us) server_negotiation.cc:1035] Waiting for connection context
0619 16:43:15.068275 (+ 400us) server_negotiation.cc:299] Negotiation successful
0619 16:43:15.068680 (+ 405us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":562,"thread_start_us":348,"threads_started":1}
I20260619 16:43:15.069921 1556 raft_consensus.cc:388] T 00000000000000000000000000000000 P de31151854fc4a318c6bc1f46f6b59bf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:43:15.070040 1556 raft_consensus.cc:482] T 00000000000000000000000000000000 P de31151854fc4a318c6bc1f46f6b59bf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:43:15.070216 1556 raft_consensus.cc:3037] T 00000000000000000000000000000000 P de31151854fc4a318c6bc1f46f6b59bf [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:43:15.072331 1556 raft_consensus.cc:504] T 00000000000000000000000000000000 P de31151854fc4a318c6bc1f46f6b59bf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de31151854fc4a318c6bc1f46f6b59bf" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34657 } }
I20260619 16:43:15.072774 1556 leader_election.cc:302] T 00000000000000000000000000000000 P de31151854fc4a318c6bc1f46f6b59bf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: de31151854fc4a318c6bc1f46f6b59bf; no voters:
I20260619 16:43:15.073257 1556 leader_election.cc:288] T 00000000000000000000000000000000 P de31151854fc4a318c6bc1f46f6b59bf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 16:43:15.073359 1561 raft_consensus.cc:2781] T 00000000000000000000000000000000 P de31151854fc4a318c6bc1f46f6b59bf [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:43:15.074353 1561 raft_consensus.cc:686] T 00000000000000000000000000000000 P de31151854fc4a318c6bc1f46f6b59bf [term 1 LEADER]: Becoming Leader. State: Replica: de31151854fc4a318c6bc1f46f6b59bf, State: Running, Role: LEADER
I20260619 16:43:15.074934 1561 consensus_queue.cc:237] T 00000000000000000000000000000000 P de31151854fc4a318c6bc1f46f6b59bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de31151854fc4a318c6bc1f46f6b59bf" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34657 } }
I20260619 16:43:15.075119 1556 sys_catalog.cc:564] T 00000000000000000000000000000000 P de31151854fc4a318c6bc1f46f6b59bf [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:43:15.077411 1563 sys_catalog.cc:455] T 00000000000000000000000000000000 P de31151854fc4a318c6bc1f46f6b59bf [sys.catalog]: SysCatalogTable state changed. Reason: New leader de31151854fc4a318c6bc1f46f6b59bf. Latest consensus state: current_term: 1 leader_uuid: "de31151854fc4a318c6bc1f46f6b59bf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de31151854fc4a318c6bc1f46f6b59bf" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34657 } } }
I20260619 16:43:15.077777 1563 sys_catalog.cc:458] T 00000000000000000000000000000000 P de31151854fc4a318c6bc1f46f6b59bf [sys.catalog]: This master's current role is: LEADER
I20260619 16:43:15.078310 1562 sys_catalog.cc:455] T 00000000000000000000000000000000 P de31151854fc4a318c6bc1f46f6b59bf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "de31151854fc4a318c6bc1f46f6b59bf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de31151854fc4a318c6bc1f46f6b59bf" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34657 } } }
I20260619 16:43:15.078572 1562 sys_catalog.cc:458] T 00000000000000000000000000000000 P de31151854fc4a318c6bc1f46f6b59bf [sys.catalog]: This master's current role is: LEADER
I20260619 16:43:15.082686 1567 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:43:15.088410 1567 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:43:15.096347 1567 catalog_manager.cc:1347] Generated new cluster ID: 70cc8cb239e14e77862e438466b21e20
I20260619 16:43:15.096444 1567 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:43:15.140475 1567 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 16:43:15.141708 1567 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:43:15.150271 1567 catalog_manager.cc:5868] T 00000000000000000000000000000000 P de31151854fc4a318c6bc1f46f6b59bf: Generated new TSK 0
I20260619 16:43:15.151120 1567 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887385903558-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887385903558-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260619 16:43:15.192569 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu
/tmp/dist-test-taskEs2EH8/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887385903558-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887385903558-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887385903558-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887385903558-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34657
--keytab_file=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887385903558-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38449
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887385903558-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887385903558-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887385903558-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:43:15.510128 1582 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:43:15.510388 1582 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:43:15.510517 1582 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:43:15.517366 1582 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:43:15.517468 1582 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:43:15.517585 1582 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 16:43:15.517683 1582 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260619 16:43:15.527164 1582 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38449
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887385903558-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887385903558-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887385903558-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887385903558-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:34657
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887385903558-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:43:15.528388 1582 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:43:15.529587 1582 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:43:15.542217 1587 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:43:15.542186 1590 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:43:15.542268 1588 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:43:15.542695 1582 server_base.cc:1034] running on GCE node
I20260619 16:43:15.543280 1582 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:43:15.544166 1582 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:43:15.545400 1582 hybrid_clock.cc:648] HybridClock initialized: now 1781887395545329 us; error 41 us; skew 500 ppm
I20260619 16:43:15.545698 1582 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:43:15 dist-test-slave-wvz7 krb5kdc[1466](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887395, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:43:15.548781 1582 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260619 16:43:15.550230 1582 webserver.cc:460] Webserver started at http://127.0.97.1:40793/ using document root <none> and password file <none>
I20260619 16:43:15.550930 1582 fs_manager.cc:359] Metadata directory not provided
I20260619 16:43:15.551028 1582 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:43:15.551326 1582 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:43:15.554180 1582 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887385903558-388-0/minicluster-data/ts-0/data/instance:
uuid: "560a8f774fe846b4b173a721079b3830"
format_stamp: "Formatted at 2026-06-19 16:43:15 on dist-test-slave-wvz7"
server_key: "753f732602bb995395e36612e5efb172"
server_key_iv: "eab86e002c3ea61446484fec7dca1f99"
server_key_version: "encryptionkey@0"
I20260619 16:43:15.554944 1582 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887385903558-388-0/minicluster-data/ts-0/wal/instance:
uuid: "560a8f774fe846b4b173a721079b3830"
format_stamp: "Formatted at 2026-06-19 16:43:15 on dist-test-slave-wvz7"
server_key: "753f732602bb995395e36612e5efb172"
server_key_iv: "eab86e002c3ea61446484fec7dca1f99"
server_key_version: "encryptionkey@0"
I20260619 16:43:15.560014 1582 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 16:43:15.563369 1597 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:43:15.564707 1582 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 16:43:15.564881 1582 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887385903558-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887385903558-388-0/minicluster-data/ts-0/wal
uuid: "560a8f774fe846b4b173a721079b3830"
format_stamp: "Formatted at 2026-06-19 16:43:15 on dist-test-slave-wvz7"
server_key: "753f732602bb995395e36612e5efb172"
server_key_iv: "eab86e002c3ea61446484fec7dca1f99"
server_key_version: "encryptionkey@0"
I20260619 16:43:15.565068 1582 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887385903558-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887385903558-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887385903558-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:43:15.610714 1582 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:43:15.615105 1582 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:43:15.615365 1582 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:43:15.617933 1582 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:43:15.618026 1582 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:43:15.618146 1582 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:43:15.618216 1582 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 16:43:15 dist-test-slave-wvz7 krb5kdc[1466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887395, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 16:43:15.640707 1606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:15.618646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:56829 (local address 127.0.97.62:34657)
0619 16:43:15.619063 (+ 417us) server_negotiation.cc:206] Beginning negotiation
0619 16:43:15.619068 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:43:15.620347 (+ 1279us) server_negotiation.cc:407] Connection header received
0619 16:43:15.622354 (+ 2007us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:43:15.622359 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:43:15.622443 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:43:15.622534 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:43:15.624252 (+ 1718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:15.625179 (+ 927us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:15.628121 (+ 2942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:15.628651 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:15.628798 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:43:15.633859 (+ 5061us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:43:15.633866 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:43:15.633871 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:43:15.633908 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:43:15.636997 (+ 3089us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:43:15.637703 (+ 706us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:43:15.637709 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:43:15.637713 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:43:15.637894 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:43:15.638509 (+ 615us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:43:15.638515 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:43:15.638519 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:43:15.638887 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:43:15.639057 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0619 16:43:15.640072 (+ 1015us) server_negotiation.cc:299] Negotiation successful
0619 16:43:15.640307 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":311,"thread_start_us":222,"threads_started":1}
I20260619 16:43:15.641840 1607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:15.619375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34657 (local address 127.0.97.1:56829)
0619 16:43:15.620179 (+ 804us) negotiation.cc:107] Waiting for socket to connect
0619 16:43:15.620238 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0619 16:43:15.622074 (+ 1836us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:43:15.622781 (+ 707us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:43:15.622792 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:43:15.623379 (+ 587us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:43:15.624046 (+ 667us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:43:15.624063 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:43:15.625434 (+ 1371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:43:15.625442 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:43:15.627955 (+ 2513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:43:15.627974 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:43:15.628883 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:43:15.628890 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:43:15.629200 (+ 310us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:43:15.630149 (+ 949us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:43:15.630194 (+ 45us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:43:15.633559 (+ 3365us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:43:15.637145 (+ 3586us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:43:15.637154 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:43:15.637173 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:43:15.637585 (+ 412us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:43:15.638030 (+ 445us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:43:15.638036 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:43:15.638041 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:43:15.638336 (+ 295us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:43:15.639036 (+ 700us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:43:15.639045 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:43:15.639561 (+ 516us) client_negotiation.cc:769] Sending connection context
0619 16:43:15.639946 (+ 385us) client_negotiation.cc:240] Negotiation successful
0619 16:43:15.640533 (+ 587us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":578,"thread_start_us":381,"threads_started":1}
I20260619 16:43:15.661900 1582 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:38031
I20260619 16:43:15.661931 1717 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:38031 every 8 connection(s)
I20260619 16:43:15.661996 1688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:15.648158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34657 (local address 127.0.97.1:51233)
0619 16:43:15.648882 (+ 724us) negotiation.cc:107] Waiting for socket to connect
0619 16:43:15.648902 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 16:43:15.649098 (+ 196us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:43:15.649638 (+ 540us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:43:15.649641 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:43:15.650302 (+ 661us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:43:15.650628 (+ 326us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:43:15.650637 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:43:15.652167 (+ 1530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:43:15.652170 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:43:15.653409 (+ 1239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:43:15.653420 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:43:15.654412 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:43:15.654421 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:43:15.654620 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:43:15.655320 (+ 700us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:43:15.655348 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:43:15.656392 (+ 1044us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:43:15.659380 (+ 2988us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:43:15.659383 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:43:15.659386 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:43:15.659925 (+ 539us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:43:15.660264 (+ 339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:43:15.660268 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:43:15.660272 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:43:15.660392 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:43:15.660840 (+ 448us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:43:15.660845 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:43:15.661015 (+ 170us) client_negotiation.cc:769] Sending connection context
0619 16:43:15.661143 (+ 128us) client_negotiation.cc:240] Negotiation successful
0619 16:43:15.661472 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":546,"thread_start_us":408,"threads_started":1}
I20260619 16:43:15.662585 1606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:15.648161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:51233 (local address 127.0.97.62:34657)
0619 16:43:15.648335 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0619 16:43:15.648342 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:43:15.648979 (+ 637us) server_negotiation.cc:407] Connection header received
0619 16:43:15.649411 (+ 432us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:43:15.649418 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:43:15.649499 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:43:15.649599 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:43:15.650759 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:15.652024 (+ 1265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:15.653549 (+ 1525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:15.654189 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:15.654336 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:43:15.656556 (+ 2220us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:43:15.656563 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:43:15.656568 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:43:15.656599 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:43:15.659235 (+ 2636us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:43:15.660028 (+ 793us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:43:15.660033 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:43:15.660037 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:43:15.660161 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:43:15.660482 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:43:15.660487 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:43:15.660491 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:43:15.660734 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:43:15.660863 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0619 16:43:15.661251 (+ 388us) server_negotiation.cc:299] Negotiation successful
0619 16:43:15.661444 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260619 16:43:15.663422 1582 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887385903558-388-0/minicluster-data/ts-0/data/info.pb
I20260619 16:43:15.665625 1718 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34657
I20260619 16:43:15.665881 1718 heartbeater.cc:464] Registering TS with master...
I20260619 16:43:15.666460 1718 heartbeater.cc:511] Master 127.0.97.62:34657 requested a full tablet report, sending...
I20260619 16:43:15.668360 1511 ts_manager.cc:194] Registered new tserver with Master: 560a8f774fe846b4b173a721079b3830 (127.0.97.1:38031)
I20260619 16:43:15.669839 1511 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:56829
I20260619 16:43:15.670818 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu as pid 1582
I20260619 16:43:15.671022 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887385903558-388-0/minicluster-data/ts-0/wal/instance
I20260619 16:43:15.671770 388 external_mini_cluster.cc:1428] Setting key 5f15590c2891b379bfc94c38cfc59b58
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887385903558-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887385903558-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260619 16:43:15.706074 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu
/tmp/dist-test-taskEs2EH8/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887385903558-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887385903558-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887385903558-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887385903558-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34657
--keytab_file=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887385903558-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38449
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887385903558-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887385903558-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887385903558-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:43:16.004428 1722 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:43:16.004701 1722 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:43:16.004805 1722 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:43:16.012015 1722 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:43:16.012121 1722 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:43:16.012215 1722 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 16:43:16.012298 1722 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260619 16:43:16.021862 1722 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38449
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887385903558-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887385903558-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887385903558-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887385903558-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:34657
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887385903558-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:43:16.023026 1722 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:43:16.024259 1722 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:43:16.036273 1727 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:43:16.036240 1730 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:43:16.036245 1728 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:43:16.036713 1722 server_base.cc:1034] running on GCE node
I20260619 16:43:16.037462 1722 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:43:16.038372 1722 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:43:16.039577 1722 hybrid_clock.cc:648] HybridClock initialized: now 1781887396039506 us; error 52 us; skew 500 ppm
I20260619 16:43:16.039849 1722 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:43:16 dist-test-slave-wvz7 krb5kdc[1466](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887396, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:43:16.042690 1722 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260619 16:43:16.044142 1722 webserver.cc:460] Webserver started at http://127.0.97.2:42403/ using document root <none> and password file <none>
I20260619 16:43:16.044817 1722 fs_manager.cc:359] Metadata directory not provided
I20260619 16:43:16.044899 1722 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:43:16.045215 1722 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:43:16.047892 1722 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887385903558-388-0/minicluster-data/ts-1/data/instance:
uuid: "3d658996110e4a3c83f948c468938fd3"
format_stamp: "Formatted at 2026-06-19 16:43:16 on dist-test-slave-wvz7"
server_key: "f1e4d19da911d8d4f0eef46b08d84759"
server_key_iv: "1acda423571ce8dbc81cd35991a57937"
server_key_version: "encryptionkey@0"
I20260619 16:43:16.048609 1722 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887385903558-388-0/minicluster-data/ts-1/wal/instance:
uuid: "3d658996110e4a3c83f948c468938fd3"
format_stamp: "Formatted at 2026-06-19 16:43:16 on dist-test-slave-wvz7"
server_key: "f1e4d19da911d8d4f0eef46b08d84759"
server_key_iv: "1acda423571ce8dbc81cd35991a57937"
server_key_version: "encryptionkey@0"
I20260619 16:43:16.053442 1722 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.003s
I20260619 16:43:16.056699 1737 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:43:16.058018 1722 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 16:43:16.058146 1722 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887385903558-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887385903558-388-0/minicluster-data/ts-1/wal
uuid: "3d658996110e4a3c83f948c468938fd3"
format_stamp: "Formatted at 2026-06-19 16:43:16 on dist-test-slave-wvz7"
server_key: "f1e4d19da911d8d4f0eef46b08d84759"
server_key_iv: "1acda423571ce8dbc81cd35991a57937"
server_key_version: "encryptionkey@0"
I20260619 16:43:16.058333 1722 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887385903558-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887385903558-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887385903558-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:43:16.079360 1722 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:43:16.082953 1722 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:43:16.083204 1722 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:43:16.085577 1722 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:43:16.085670 1722 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:43:16.085812 1722 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:43:16.085894 1722 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 16:43:16 dist-test-slave-wvz7 krb5kdc[1466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887396, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 16:43:16.107864 1606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:16.086297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:38685 (local address 127.0.97.62:34657)
0619 16:43:16.086530 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0619 16:43:16.086536 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:43:16.087837 (+ 1301us) server_negotiation.cc:407] Connection header received
0619 16:43:16.089903 (+ 2066us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:43:16.089909 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:43:16.089990 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:43:16.090098 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:43:16.092176 (+ 2078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:16.093125 (+ 949us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:16.095452 (+ 2327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:16.096007 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:16.096136 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:43:16.100786 (+ 4650us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:43:16.100794 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:43:16.100800 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:43:16.100848 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:43:16.104272 (+ 3424us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:43:16.105037 (+ 765us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:43:16.105044 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:43:16.105047 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:43:16.105204 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:43:16.105804 (+ 600us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:43:16.105809 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:43:16.105812 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:43:16.106194 (+ 382us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:43:16.106337 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0619 16:43:16.107309 (+ 972us) server_negotiation.cc:299] Negotiation successful
0619 16:43:16.107536 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136}
I20260619 16:43:16.109436 1746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:16.086936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34657 (local address 127.0.97.2:38685)
0619 16:43:16.087695 (+ 759us) negotiation.cc:107] Waiting for socket to connect
0619 16:43:16.087738 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0619 16:43:16.089599 (+ 1861us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:43:16.090472 (+ 873us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:43:16.090483 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:43:16.091216 (+ 733us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:43:16.092006 (+ 790us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:43:16.092024 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:43:16.093290 (+ 1266us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:43:16.093296 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:43:16.095252 (+ 1956us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:43:16.095264 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:43:16.096305 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:43:16.096312 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:43:16.096489 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:43:16.097163 (+ 674us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:43:16.097200 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:43:16.100482 (+ 3282us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:43:16.104448 (+ 3966us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:43:16.104456 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:43:16.104471 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:43:16.104920 (+ 449us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:43:16.105327 (+ 407us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:43:16.105333 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:43:16.105337 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:43:16.105625 (+ 288us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:43:16.106337 (+ 712us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:43:16.106346 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:43:16.106841 (+ 495us) client_negotiation.cc:769] Sending connection context
0619 16:43:16.107192 (+ 351us) client_negotiation.cc:240] Negotiation successful
0619 16:43:16.107787 (+ 595us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":533,"thread_start_us":336,"threads_started":1}
I20260619 16:43:16.125536 1722 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:44995
I20260619 16:43:16.125541 1856 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:44995 every 8 connection(s)
I20260619 16:43:16.126873 1722 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887385903558-388-0/minicluster-data/ts-1/data/info.pb
I20260619 16:43:16.128844 1857 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34657
I20260619 16:43:16.129060 1857 heartbeater.cc:464] Registering TS with master...
I20260619 16:43:16.129151 1606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:16.116157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:50043 (local address 127.0.97.62:34657)
0619 16:43:16.116311 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0619 16:43:16.116317 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:43:16.116783 (+ 466us) server_negotiation.cc:407] Connection header received
0619 16:43:16.116935 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:43:16.116941 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:43:16.117007 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:43:16.117094 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:43:16.118359 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:16.119194 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:16.121005 (+ 1811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:16.121549 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:16.121661 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:43:16.124017 (+ 2356us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:43:16.124021 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:43:16.124025 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:43:16.124052 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:43:16.126539 (+ 2487us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:43:16.127221 (+ 682us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:43:16.127226 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:43:16.127228 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:43:16.127353 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:43:16.127787 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:43:16.127792 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:43:16.127795 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:43:16.128184 (+ 389us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:43:16.128300 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0619 16:43:16.128643 (+ 343us) server_negotiation.cc:299] Negotiation successful
0619 16:43:16.128861 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260619 16:43:16.129287 1829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:16.116160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34657 (local address 127.0.97.2:50043)
0619 16:43:16.116695 (+ 535us) negotiation.cc:107] Waiting for socket to connect
0619 16:43:16.116709 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 16:43:16.116835 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:43:16.117262 (+ 427us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:43:16.117267 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:43:16.117916 (+ 649us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:43:16.118230 (+ 314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:43:16.118240 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:43:16.119313 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:43:16.119319 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:43:16.120828 (+ 1509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:43:16.120840 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:43:16.121880 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:43:16.121884 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:43:16.121987 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:43:16.122581 (+ 594us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:43:16.122606 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:43:16.123852 (+ 1246us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:43:16.126652 (+ 2800us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:43:16.126657 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:43:16.126660 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:43:16.127122 (+ 462us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:43:16.127467 (+ 345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:43:16.127475 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:43:16.127478 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:43:16.127634 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:43:16.128289 (+ 655us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:43:16.128294 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:43:16.128436 (+ 142us) client_negotiation.cc:769] Sending connection context
0619 16:43:16.128569 (+ 133us) client_negotiation.cc:240] Negotiation successful
0619 16:43:16.128868 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":422,"thread_start_us":302,"threads_started":1}
I20260619 16:43:16.129859 1857 heartbeater.cc:511] Master 127.0.97.62:34657 requested a full tablet report, sending...
I20260619 16:43:16.130422 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu as pid 1722
I20260619 16:43:16.130595 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887385903558-388-0/minicluster-data/ts-1/wal/instance
I20260619 16:43:16.131040 388 external_mini_cluster.cc:1428] Setting key dbcefbb7833bf2fedac4de4122f26d73
I20260619 16:43:16.131234 1511 ts_manager.cc:194] Registered new tserver with Master: 3d658996110e4a3c83f948c468938fd3 (127.0.97.2:44995)
I20260619 16:43:16.132253 1511 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:38685
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887385903558-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887385903558-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260619 16:43:16.163952 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu
/tmp/dist-test-taskEs2EH8/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887385903558-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887385903558-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887385903558-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887385903558-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34657
--keytab_file=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887385903558-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38449
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887385903558-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887385903558-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887385903558-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:43:16.464257 1861 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:43:16.464519 1861 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:43:16.464643 1861 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:43:16.471801 1861 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:43:16.471906 1861 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:43:16.472021 1861 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 16:43:16.472165 1861 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260619 16:43:16.481678 1861 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38449
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887385903558-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887385903558-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887385903558-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887385903558-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:34657
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887385903558-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:43:16.482879 1861 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:43:16.484176 1861 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:43:16.496198 1866 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:43:16.496308 1867 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:43:16.496228 1869 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:43:16.497014 1861 server_base.cc:1034] running on GCE node
I20260619 16:43:16.497653 1861 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:43:16.498549 1861 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:43:16.499790 1861 hybrid_clock.cc:648] HybridClock initialized: now 1781887396499725 us; error 43 us; skew 500 ppm
I20260619 16:43:16.500088 1861 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:43:16 dist-test-slave-wvz7 krb5kdc[1466](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887396, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:43:16.503149 1861 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260619 16:43:16.504541 1861 webserver.cc:460] Webserver started at http://127.0.97.3:43465/ using document root <none> and password file <none>
I20260619 16:43:16.505183 1861 fs_manager.cc:359] Metadata directory not provided
I20260619 16:43:16.505263 1861 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:43:16.505512 1861 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:43:16.508253 1861 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887385903558-388-0/minicluster-data/ts-2/data/instance:
uuid: "24c8c38e712041bca6609c1d45165595"
format_stamp: "Formatted at 2026-06-19 16:43:16 on dist-test-slave-wvz7"
server_key: "13b56ca1cbb9210e98417c34ebcdf2dc"
server_key_iv: "542b294df8d0eb9501c76480b93e3b28"
server_key_version: "encryptionkey@0"
I20260619 16:43:16.508951 1861 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887385903558-388-0/minicluster-data/ts-2/wal/instance:
uuid: "24c8c38e712041bca6609c1d45165595"
format_stamp: "Formatted at 2026-06-19 16:43:16 on dist-test-slave-wvz7"
server_key: "13b56ca1cbb9210e98417c34ebcdf2dc"
server_key_iv: "542b294df8d0eb9501c76480b93e3b28"
server_key_version: "encryptionkey@0"
I20260619 16:43:16.513860 1861 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260619 16:43:16.517227 1876 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:43:16.518664 1861 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 16:43:16.518802 1861 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887385903558-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887385903558-388-0/minicluster-data/ts-2/wal
uuid: "24c8c38e712041bca6609c1d45165595"
format_stamp: "Formatted at 2026-06-19 16:43:16 on dist-test-slave-wvz7"
server_key: "13b56ca1cbb9210e98417c34ebcdf2dc"
server_key_iv: "542b294df8d0eb9501c76480b93e3b28"
server_key_version: "encryptionkey@0"
I20260619 16:43:16.518973 1861 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887385903558-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887385903558-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887385903558-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:43:16.534051 1861 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:43:16.538079 1861 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:43:16.538290 1861 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:43:16.540887 1861 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:43:16.540987 1861 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:43:16.541126 1861 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:43:16.541257 1861 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
Jun 19 16:43:16 dist-test-slave-wvz7 krb5kdc[1466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887396, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 16:43:16.561966 1606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:16.541283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:56469 (local address 127.0.97.62:34657)
0619 16:43:16.541443 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0619 16:43:16.541448 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:43:16.542895 (+ 1447us) server_negotiation.cc:407] Connection header received
0619 16:43:16.544859 (+ 1964us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:43:16.544865 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:43:16.544947 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:43:16.545031 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:43:16.547261 (+ 2230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:16.548448 (+ 1187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:16.550715 (+ 2267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:16.551338 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:16.551466 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:43:16.555410 (+ 3944us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:43:16.555418 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:43:16.555424 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:43:16.555472 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:43:16.558816 (+ 3344us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:43:16.559552 (+ 736us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:43:16.559559 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:43:16.559563 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:43:16.559742 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:43:16.560109 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:43:16.560114 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:43:16.560118 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:43:16.560464 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:43:16.560606 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0619 16:43:16.561327 (+ 721us) server_negotiation.cc:299] Negotiation successful
0619 16:43:16.561548 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260619 16:43:16.563325 1885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:16.541984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34657 (local address 127.0.97.3:56469)
0619 16:43:16.542731 (+ 747us) negotiation.cc:107] Waiting for socket to connect
0619 16:43:16.542780 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0619 16:43:16.544565 (+ 1785us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:43:16.545324 (+ 759us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:43:16.545336 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:43:16.546083 (+ 747us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:43:16.547026 (+ 943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:43:16.547049 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:43:16.548613 (+ 1564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:43:16.548617 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:43:16.550558 (+ 1941us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:43:16.550574 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:43:16.551565 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:43:16.551575 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:43:16.551844 (+ 269us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:43:16.552495 (+ 651us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:43:16.552520 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:43:16.555152 (+ 2632us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:43:16.558987 (+ 3835us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:43:16.558993 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:43:16.559007 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:43:16.559415 (+ 408us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:43:16.559841 (+ 426us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:43:16.559844 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:43:16.559846 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:43:16.560016 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:43:16.560562 (+ 546us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:43:16.560567 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:43:16.560891 (+ 324us) client_negotiation.cc:769] Sending connection context
0619 16:43:16.561175 (+ 284us) client_negotiation.cc:240] Negotiation successful
0619 16:43:16.561797 (+ 622us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":520,"thread_start_us":346,"threads_started":1}
I20260619 16:43:16.584306 1606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:16.569281 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:45871 (local address 127.0.97.62:34657)
0619 16:43:16.569449 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0619 16:43:16.569455 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:43:16.570263 (+ 808us) server_negotiation.cc:407] Connection header received
0619 16:43:16.570436 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:43:16.570441 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:43:16.570515 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:43:16.570606 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:43:16.571948 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:16.572997 (+ 1049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:16.575188 (+ 2191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:16.575783 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:16.575908 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:43:16.578203 (+ 2295us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:43:16.578209 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:43:16.578213 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:43:16.578252 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:43:16.581264 (+ 3012us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:43:16.582259 (+ 995us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:43:16.582265 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:43:16.582269 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:43:16.582418 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:43:16.582862 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:43:16.582867 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:43:16.582871 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:43:16.583117 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:43:16.583234 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0619 16:43:16.583717 (+ 483us) server_negotiation.cc:299] Negotiation successful
0619 16:43:16.583950 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260619 16:43:16.584468 1948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:16.569380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34657 (local address 127.0.97.3:45871)
0619 16:43:16.570173 (+ 793us) negotiation.cc:107] Waiting for socket to connect
0619 16:43:16.570189 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 16:43:16.570317 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:43:16.570680 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:43:16.570686 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:43:16.571316 (+ 630us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:43:16.571801 (+ 485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:43:16.571815 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:43:16.573167 (+ 1352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:43:16.573172 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:43:16.575043 (+ 1871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:43:16.575060 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:43:16.576009 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:43:16.576015 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:43:16.576207 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:43:16.576815 (+ 608us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:43:16.576842 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:43:16.578014 (+ 1172us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:43:16.581407 (+ 3393us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:43:16.581416 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:43:16.581420 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:43:16.582074 (+ 654us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:43:16.582538 (+ 464us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:43:16.582543 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:43:16.582546 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:43:16.582731 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:43:16.583228 (+ 497us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:43:16.583232 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:43:16.583421 (+ 189us) client_negotiation.cc:769] Sending connection context
0619 16:43:16.583577 (+ 156us) client_negotiation.cc:240] Negotiation successful
0619 16:43:16.583951 (+ 374us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":677,"thread_start_us":436,"threads_started":1}
I20260619 16:43:16.584986 1861 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:42937
I20260619 16:43:16.585026 1995 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:42937 every 8 connection(s)
I20260619 16:43:16.586532 1861 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887385903558-388-0/minicluster-data/ts-2/data/info.pb
I20260619 16:43:16.588397 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu as pid 1861
I20260619 16:43:16.588568 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887385903558-388-0/minicluster-data/ts-2/wal/instance
I20260619 16:43:16.588936 388 external_mini_cluster.cc:1428] Setting key 399f468be1930b24b26b561ec1e7d8f6
I20260619 16:43:16.589625 1996 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34657
I20260619 16:43:16.589872 1996 heartbeater.cc:464] Registering TS with master...
I20260619 16:43:16.590551 1996 heartbeater.cc:511] Master 127.0.97.62:34657 requested a full tablet report, sending...
I20260619 16:43:16.591699 1510 ts_manager.cc:194] Registered new tserver with Master: 24c8c38e712041bca6609c1d45165595 (127.0.97.3:42937)
I20260619 16:43:16.592478 1510 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:56469
I20260619 16:43:16.603525 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 19 16:43:16 dist-test-slave-wvz7 krb5kdc[1466](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887396, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 19 16:43:16 dist-test-slave-wvz7 krb5kdc[1466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887396, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 16:43:16.644320 1606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:16.631207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40224 (local address 127.0.97.62:34657)
0619 16:43:16.631352 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0619 16:43:16.631356 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:43:16.631684 (+ 328us) server_negotiation.cc:407] Connection header received
0619 16:43:16.631812 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:43:16.631818 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:43:16.631907 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:43:16.632000 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:43:16.633374 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:16.634191 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:16.635496 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:16.635922 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:16.636026 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:43:16.639724 (+ 3698us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:43:16.639729 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:43:16.639732 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:43:16.639766 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:43:16.642048 (+ 2282us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:43:16.642612 (+ 564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:43:16.642615 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:43:16.642618 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:43:16.642727 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:43:16.643141 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:43:16.643146 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:43:16.643148 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:43:16.643411 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:43:16.643641 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0619 16:43:16.643888 (+ 247us) server_negotiation.cc:299] Negotiation successful
0619 16:43:16.644049 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260619 16:43:16.648504 1510 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40224:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 16:43:16.655037 1510 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 16:43:16.672875 1718 heartbeater.cc:503] Master 127.0.97.62:34657 was elected leader, sending a full tablet report...
I20260619 16:43:16.694901 2011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:16.667751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49556 (local address 127.0.97.2:44995)
0619 16:43:16.686193 (+ 18442us) server_negotiation.cc:206] Beginning negotiation
0619 16:43:16.686206 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 16:43:16.686246 (+ 40us) server_negotiation.cc:407] Connection header received
0619 16:43:16.686370 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:43:16.686381 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:43:16.686720 (+ 339us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:43:16.687000 (+ 280us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:43:16.687849 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:16.690391 (+ 2542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:16.692617 (+ 2226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:16.693649 (+ 1032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:16.693812 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:43:16.694027 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0619 16:43:16.694412 (+ 385us) server_negotiation.cc:299] Negotiation successful
0619 16:43:16.694590 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18271,"thread_start_us":18130,"threads_started":1}
I20260619 16:43:16.694953 2010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:16.667436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:44995 (local address 127.0.0.1:49556)
0619 16:43:16.668414 (+ 978us) negotiation.cc:107] Waiting for socket to connect
0619 16:43:16.668468 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0619 16:43:16.668634 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:43:16.687031 (+ 18397us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:43:16.687041 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:43:16.687082 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:43:16.687595 (+ 513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:43:16.687607 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:43:16.690531 (+ 2924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:43:16.690537 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:43:16.692450 (+ 1913us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:43:16.692462 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:43:16.693885 (+ 1423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:43:16.693892 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:43:16.694127 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:43:16.694189 (+ 62us) client_negotiation.cc:769] Sending connection context
0619 16:43:16.694346 (+ 157us) client_negotiation.cc:240] Negotiation successful
0619 16:43:16.694537 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":634,"thread_start_us":21888,"threads_started":1}
I20260619 16:43:16.695887 2013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:16.667560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:38031 (local address 127.0.0.1:60058)
0619 16:43:16.682699 (+ 15139us) negotiation.cc:107] Waiting for socket to connect
0619 16:43:16.682723 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0619 16:43:16.682865 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:43:16.688527 (+ 5662us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:43:16.688532 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:43:16.688557 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:43:16.689029 (+ 472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:43:16.689055 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:43:16.690985 (+ 1930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:43:16.690991 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:43:16.693275 (+ 2284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:43:16.693290 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:43:16.695072 (+ 1782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:43:16.695079 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:43:16.695256 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:43:16.695310 (+ 54us) client_negotiation.cc:769] Sending connection context
0619 16:43:16.695378 (+ 68us) client_negotiation.cc:240] Negotiation successful
0619 16:43:16.695512 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14989,"thread_start_us":368,"threads_started":1}
I20260619 16:43:16.696998 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:16.668158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60058 (local address 127.0.97.1:38031)
0619 16:43:16.687618 (+ 19460us) server_negotiation.cc:206] Beginning negotiation
0619 16:43:16.687627 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 16:43:16.687656 (+ 29us) server_negotiation.cc:407] Connection header received
0619 16:43:16.687756 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:43:16.687764 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:43:16.688287 (+ 523us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:43:16.689252 (+ 965us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:43:16.689472 (+ 220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:16.690808 (+ 1336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:16.693428 (+ 2620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:16.694817 (+ 1389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:16.696224 (+ 1407us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:43:16.696453 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0619 16:43:16.696523 (+ 70us) server_negotiation.cc:299] Negotiation successful
0619 16:43:16.696701 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19315,"thread_start_us":19193,"threads_started":1}
I20260619 16:43:16.698702 2009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:16.669232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:42937 (local address 127.0.0.1:57588)
0619 16:43:16.689685 (+ 20453us) negotiation.cc:107] Waiting for socket to connect
0619 16:43:16.689712 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0619 16:43:16.689882 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:43:16.690800 (+ 918us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:43:16.690805 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:43:16.690830 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:43:16.691264 (+ 434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:43:16.691277 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:43:16.693315 (+ 2038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:43:16.693320 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:43:16.695607 (+ 2287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:43:16.695620 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:43:16.697919 (+ 2299us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:43:16.697925 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:43:16.698091 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:43:16.698134 (+ 43us) client_negotiation.cc:769] Sending connection context
0619 16:43:16.698223 (+ 89us) client_negotiation.cc:240] Negotiation successful
0619 16:43:16.698347 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":20256,"mutex_wait_us":1,"thread_start_us":13829,"threads_started":1}
I20260619 16:43:16.698817 2014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:16.669517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57588 (local address 127.0.97.3:42937)
0619 16:43:16.681970 (+ 12453us) server_negotiation.cc:206] Beginning negotiation
0619 16:43:16.681981 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 16:43:16.689868 (+ 7887us) server_negotiation.cc:407] Connection header received
0619 16:43:16.690024 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:43:16.690031 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:43:16.690260 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:43:16.690460 (+ 200us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:43:16.691514 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:16.693157 (+ 1643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:16.695758 (+ 2601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:16.697425 (+ 1667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:16.697505 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:43:16.697660 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0619 16:43:16.698290 (+ 630us) server_negotiation.cc:299] Negotiation successful
0619 16:43:16.698466 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12234,"thread_start_us":12484,"threads_started":1}
I20260619 16:43:16.699538 1787 tablet_service.cc:1467] Processing CreateTablet for tablet 64d666c55f4a4b56ad385718f4fc5209 (DEFAULT_TABLE table=test-table [id=1b75f4cef8ab47b18cdd875fbaa3688b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:43:16.700832 1787 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 64d666c55f4a4b56ad385718f4fc5209. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:43:16.700326 1648 tablet_service.cc:1467] Processing CreateTablet for tablet 64d666c55f4a4b56ad385718f4fc5209 (DEFAULT_TABLE table=test-table [id=1b75f4cef8ab47b18cdd875fbaa3688b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:43:16.701637 1648 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 64d666c55f4a4b56ad385718f4fc5209. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:43:16.702272 1926 tablet_service.cc:1467] Processing CreateTablet for tablet 64d666c55f4a4b56ad385718f4fc5209 (DEFAULT_TABLE table=test-table [id=1b75f4cef8ab47b18cdd875fbaa3688b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:43:16.703552 1926 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 64d666c55f4a4b56ad385718f4fc5209. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:43:16.711869 2015 tablet_bootstrap.cc:492] T 64d666c55f4a4b56ad385718f4fc5209 P 3d658996110e4a3c83f948c468938fd3: Bootstrap starting.
I20260619 16:43:16.714944 2015 tablet_bootstrap.cc:654] T 64d666c55f4a4b56ad385718f4fc5209 P 3d658996110e4a3c83f948c468938fd3: Neither blocks nor log segments found. Creating new log.
I20260619 16:43:16.715937 2016 tablet_bootstrap.cc:492] T 64d666c55f4a4b56ad385718f4fc5209 P 560a8f774fe846b4b173a721079b3830: Bootstrap starting.
I20260619 16:43:16.717053 2015 log.cc:824] T 64d666c55f4a4b56ad385718f4fc5209 P 3d658996110e4a3c83f948c468938fd3: Log is configured to *not* fsync() on all Append() calls
I20260619 16:43:16.718010 2017 tablet_bootstrap.cc:492] T 64d666c55f4a4b56ad385718f4fc5209 P 24c8c38e712041bca6609c1d45165595: Bootstrap starting.
I20260619 16:43:16.719573 2016 tablet_bootstrap.cc:654] T 64d666c55f4a4b56ad385718f4fc5209 P 560a8f774fe846b4b173a721079b3830: Neither blocks nor log segments found. Creating new log.
I20260619 16:43:16.720821 2016 log.cc:824] T 64d666c55f4a4b56ad385718f4fc5209 P 560a8f774fe846b4b173a721079b3830: Log is configured to *not* fsync() on all Append() calls
I20260619 16:43:16.721230 2017 tablet_bootstrap.cc:654] T 64d666c55f4a4b56ad385718f4fc5209 P 24c8c38e712041bca6609c1d45165595: Neither blocks nor log segments found. Creating new log.
I20260619 16:43:16.721310 2015 tablet_bootstrap.cc:492] T 64d666c55f4a4b56ad385718f4fc5209 P 3d658996110e4a3c83f948c468938fd3: No bootstrap required, opened a new log
I20260619 16:43:16.721644 2015 ts_tablet_manager.cc:1397] T 64d666c55f4a4b56ad385718f4fc5209 P 3d658996110e4a3c83f948c468938fd3: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260619 16:43:16.722371 2017 log.cc:824] T 64d666c55f4a4b56ad385718f4fc5209 P 24c8c38e712041bca6609c1d45165595: Log is configured to *not* fsync() on all Append() calls
I20260619 16:43:16.723757 2016 tablet_bootstrap.cc:492] T 64d666c55f4a4b56ad385718f4fc5209 P 560a8f774fe846b4b173a721079b3830: No bootstrap required, opened a new log
I20260619 16:43:16.724014 2016 ts_tablet_manager.cc:1397] T 64d666c55f4a4b56ad385718f4fc5209 P 560a8f774fe846b4b173a721079b3830: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260619 16:43:16.724699 2017 tablet_bootstrap.cc:492] T 64d666c55f4a4b56ad385718f4fc5209 P 24c8c38e712041bca6609c1d45165595: No bootstrap required, opened a new log
I20260619 16:43:16.724910 2017 ts_tablet_manager.cc:1397] T 64d666c55f4a4b56ad385718f4fc5209 P 24c8c38e712041bca6609c1d45165595: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260619 16:43:16.732004 2015 raft_consensus.cc:348] T 64d666c55f4a4b56ad385718f4fc5209 P 3d658996110e4a3c83f948c468938fd3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d658996110e4a3c83f948c468938fd3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44995 } } peers { permanent_uuid: "560a8f774fe846b4b173a721079b3830" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38031 } } peers { permanent_uuid: "24c8c38e712041bca6609c1d45165595" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42937 } }
I20260619 16:43:16.732460 2015 raft_consensus.cc:374] T 64d666c55f4a4b56ad385718f4fc5209 P 3d658996110e4a3c83f948c468938fd3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:43:16.732611 2015 raft_consensus.cc:729] T 64d666c55f4a4b56ad385718f4fc5209 P 3d658996110e4a3c83f948c468938fd3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d658996110e4a3c83f948c468938fd3, State: Initialized, Role: FOLLOWER
I20260619 16:43:16.732488 2017 raft_consensus.cc:348] T 64d666c55f4a4b56ad385718f4fc5209 P 24c8c38e712041bca6609c1d45165595 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d658996110e4a3c83f948c468938fd3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44995 } } peers { permanent_uuid: "560a8f774fe846b4b173a721079b3830" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38031 } } peers { permanent_uuid: "24c8c38e712041bca6609c1d45165595" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42937 } }
I20260619 16:43:16.732851 2017 raft_consensus.cc:374] T 64d666c55f4a4b56ad385718f4fc5209 P 24c8c38e712041bca6609c1d45165595 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:43:16.732980 2017 raft_consensus.cc:729] T 64d666c55f4a4b56ad385718f4fc5209 P 24c8c38e712041bca6609c1d45165595 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 24c8c38e712041bca6609c1d45165595, State: Initialized, Role: FOLLOWER
I20260619 16:43:16.733244 2015 consensus_queue.cc:260] T 64d666c55f4a4b56ad385718f4fc5209 P 3d658996110e4a3c83f948c468938fd3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d658996110e4a3c83f948c468938fd3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44995 } } peers { permanent_uuid: "560a8f774fe846b4b173a721079b3830" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38031 } } peers { permanent_uuid: "24c8c38e712041bca6609c1d45165595" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42937 } }
I20260619 16:43:16.733482 2017 consensus_queue.cc:260] T 64d666c55f4a4b56ad385718f4fc5209 P 24c8c38e712041bca6609c1d45165595 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d658996110e4a3c83f948c468938fd3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44995 } } peers { permanent_uuid: "560a8f774fe846b4b173a721079b3830" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38031 } } peers { permanent_uuid: "24c8c38e712041bca6609c1d45165595" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42937 } }
I20260619 16:43:16.734213 2016 raft_consensus.cc:348] T 64d666c55f4a4b56ad385718f4fc5209 P 560a8f774fe846b4b173a721079b3830 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d658996110e4a3c83f948c468938fd3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44995 } } peers { permanent_uuid: "560a8f774fe846b4b173a721079b3830" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38031 } } peers { permanent_uuid: "24c8c38e712041bca6609c1d45165595" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42937 } }
I20260619 16:43:16.734577 1996 heartbeater.cc:503] Master 127.0.97.62:34657 was elected leader, sending a full tablet report...
I20260619 16:43:16.734622 2016 raft_consensus.cc:374] T 64d666c55f4a4b56ad385718f4fc5209 P 560a8f774fe846b4b173a721079b3830 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:43:16.735169 2016 raft_consensus.cc:729] T 64d666c55f4a4b56ad385718f4fc5209 P 560a8f774fe846b4b173a721079b3830 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 560a8f774fe846b4b173a721079b3830, State: Initialized, Role: FOLLOWER
I20260619 16:43:16.735332 2017 ts_tablet_manager.cc:1428] T 64d666c55f4a4b56ad385718f4fc5209 P 24c8c38e712041bca6609c1d45165595: Time spent starting tablet: real 0.010s user 0.010s sys 0.000s
I20260619 16:43:16.735824 2016 consensus_queue.cc:260] T 64d666c55f4a4b56ad385718f4fc5209 P 560a8f774fe846b4b173a721079b3830 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d658996110e4a3c83f948c468938fd3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44995 } } peers { permanent_uuid: "560a8f774fe846b4b173a721079b3830" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38031 } } peers { permanent_uuid: "24c8c38e712041bca6609c1d45165595" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42937 } }
I20260619 16:43:16.737700 2016 ts_tablet_manager.cc:1428] T 64d666c55f4a4b56ad385718f4fc5209 P 560a8f774fe846b4b173a721079b3830: Time spent starting tablet: real 0.014s user 0.014s sys 0.000s
I20260619 16:43:16.742313 1857 heartbeater.cc:503] Master 127.0.97.62:34657 was elected leader, sending a full tablet report...
I20260619 16:43:16.742739 2015 ts_tablet_manager.cc:1428] T 64d666c55f4a4b56ad385718f4fc5209 P 3d658996110e4a3c83f948c468938fd3: Time spent starting tablet: real 0.021s user 0.008s sys 0.010s
W20260619 16:43:16.838788 1997 tablet.cc:2369] T 64d666c55f4a4b56ad385718f4fc5209 P 24c8c38e712041bca6609c1d45165595: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 16:43:16.838806 2021 raft_consensus.cc:482] T 64d666c55f4a4b56ad385718f4fc5209 P 3d658996110e4a3c83f948c468938fd3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 16:43:16.839082 2021 raft_consensus.cc:504] T 64d666c55f4a4b56ad385718f4fc5209 P 3d658996110e4a3c83f948c468938fd3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d658996110e4a3c83f948c468938fd3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44995 } } peers { permanent_uuid: "560a8f774fe846b4b173a721079b3830" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38031 } } peers { permanent_uuid: "24c8c38e712041bca6609c1d45165595" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42937 } }
I20260619 16:43:16.840597 2021 leader_election.cc:288] T 64d666c55f4a4b56ad385718f4fc5209 P 3d658996110e4a3c83f948c468938fd3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 560a8f774fe846b4b173a721079b3830 (127.0.97.1:38031), 24c8c38e712041bca6609c1d45165595 (127.0.97.3:42937)
I20260619 16:43:16.847000 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:16.840901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:60555 (local address 127.0.97.1:38031)
0619 16:43:16.841073 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0619 16:43:16.841078 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:43:16.841546 (+ 468us) server_negotiation.cc:407] Connection header received
0619 16:43:16.841825 (+ 279us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:43:16.841831 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:43:16.841907 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:43:16.841998 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:43:16.842561 (+ 563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:16.843252 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:16.845428 (+ 2176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:16.846200 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:16.846316 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:43:16.846410 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0619 16:43:16.846672 (+ 262us) server_negotiation.cc:299] Negotiation successful
0619 16:43:16.846789 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260619 16:43:16.847159 2024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:16.840851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:38031 (local address 127.0.97.2:60555)
0619 16:43:16.841444 (+ 593us) negotiation.cc:107] Waiting for socket to connect
0619 16:43:16.841463 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 16:43:16.841596 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:43:16.842060 (+ 464us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:43:16.842084 (+ 24us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:43:16.842112 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:43:16.842428 (+ 316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:43:16.842437 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:43:16.843369 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:43:16.843375 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:43:16.845292 (+ 1917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:43:16.845303 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:43:16.846386 (+ 1083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:43:16.846392 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:43:16.846534 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:43:16.846556 (+ 22us) client_negotiation.cc:769] Sending connection context
0619 16:43:16.846646 (+ 90us) client_negotiation.cc:240] Negotiation successful
0619 16:43:16.846760 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":477,"mutex_wait_us":3,"spinlock_wait_cycles":32000,"thread_start_us":279,"threads_started":1}
I20260619 16:43:16.847227 2014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:16.841801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:42513 (local address 127.0.97.3:42937)
0619 16:43:16.841976 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0619 16:43:16.841981 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:43:16.842231 (+ 250us) server_negotiation.cc:407] Connection header received
0619 16:43:16.842404 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:43:16.842407 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:43:16.842461 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:43:16.842530 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:43:16.842978 (+ 448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:16.843647 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:16.845684 (+ 2037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:16.846450 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:16.846571 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:43:16.846665 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0619 16:43:16.846893 (+ 228us) server_negotiation.cc:299] Negotiation successful
0619 16:43:16.847005 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260619 16:43:16.847460 2025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:16.841596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:42937 (local address 127.0.97.2:42513)
0619 16:43:16.842136 (+ 540us) negotiation.cc:107] Waiting for socket to connect
0619 16:43:16.842149 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 16:43:16.842287 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:43:16.842553 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:43:16.842557 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:43:16.842578 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:43:16.842851 (+ 273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:43:16.842859 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:43:16.843793 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:43:16.843797 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:43:16.845559 (+ 1762us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:43:16.845570 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:43:16.846637 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:43:16.846642 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:43:16.846764 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:43:16.846786 (+ 22us) client_negotiation.cc:769] Sending connection context
0619 16:43:16.846860 (+ 74us) client_negotiation.cc:240] Negotiation successful
0619 16:43:16.846976 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":438,"mutex_wait_us":45,"spinlock_wait_cycles":896,"thread_start_us":286,"threads_started":1}
I20260619 16:43:16.848234 1668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64d666c55f4a4b56ad385718f4fc5209" candidate_uuid: "3d658996110e4a3c83f948c468938fd3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "560a8f774fe846b4b173a721079b3830" is_pre_election: true
I20260619 16:43:16.848233 1951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64d666c55f4a4b56ad385718f4fc5209" candidate_uuid: "3d658996110e4a3c83f948c468938fd3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "24c8c38e712041bca6609c1d45165595" is_pre_election: true
I20260619 16:43:16.848748 1951 raft_consensus.cc:2454] T 64d666c55f4a4b56ad385718f4fc5209 P 24c8c38e712041bca6609c1d45165595 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d658996110e4a3c83f948c468938fd3 in term 0.
I20260619 16:43:16.848748 1668 raft_consensus.cc:2454] T 64d666c55f4a4b56ad385718f4fc5209 P 560a8f774fe846b4b173a721079b3830 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d658996110e4a3c83f948c468938fd3 in term 0.
I20260619 16:43:16.849337 1740 leader_election.cc:302] T 64d666c55f4a4b56ad385718f4fc5209 P 3d658996110e4a3c83f948c468938fd3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d658996110e4a3c83f948c468938fd3, 560a8f774fe846b4b173a721079b3830; no voters:
I20260619 16:43:16.849666 2021 raft_consensus.cc:2781] T 64d666c55f4a4b56ad385718f4fc5209 P 3d658996110e4a3c83f948c468938fd3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 16:43:16.849854 2021 raft_consensus.cc:482] T 64d666c55f4a4b56ad385718f4fc5209 P 3d658996110e4a3c83f948c468938fd3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 16:43:16.849962 2021 raft_consensus.cc:3037] T 64d666c55f4a4b56ad385718f4fc5209 P 3d658996110e4a3c83f948c468938fd3 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:43:16.852159 2021 raft_consensus.cc:504] T 64d666c55f4a4b56ad385718f4fc5209 P 3d658996110e4a3c83f948c468938fd3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d658996110e4a3c83f948c468938fd3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44995 } } peers { permanent_uuid: "560a8f774fe846b4b173a721079b3830" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38031 } } peers { permanent_uuid: "24c8c38e712041bca6609c1d45165595" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42937 } }
I20260619 16:43:16.852756 2021 leader_election.cc:288] T 64d666c55f4a4b56ad385718f4fc5209 P 3d658996110e4a3c83f948c468938fd3 [CANDIDATE]: Term 1 election: Requested vote from peers 560a8f774fe846b4b173a721079b3830 (127.0.97.1:38031), 24c8c38e712041bca6609c1d45165595 (127.0.97.3:42937)
I20260619 16:43:16.853168 1668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64d666c55f4a4b56ad385718f4fc5209" candidate_uuid: "3d658996110e4a3c83f948c468938fd3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "560a8f774fe846b4b173a721079b3830"
I20260619 16:43:16.853225 1951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64d666c55f4a4b56ad385718f4fc5209" candidate_uuid: "3d658996110e4a3c83f948c468938fd3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "24c8c38e712041bca6609c1d45165595"
I20260619 16:43:16.853340 1668 raft_consensus.cc:3037] T 64d666c55f4a4b56ad385718f4fc5209 P 560a8f774fe846b4b173a721079b3830 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:43:16.853407 1951 raft_consensus.cc:3037] T 64d666c55f4a4b56ad385718f4fc5209 P 24c8c38e712041bca6609c1d45165595 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:43:16.855422 1668 raft_consensus.cc:2454] T 64d666c55f4a4b56ad385718f4fc5209 P 560a8f774fe846b4b173a721079b3830 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d658996110e4a3c83f948c468938fd3 in term 1.
I20260619 16:43:16.855633 1951 raft_consensus.cc:2454] T 64d666c55f4a4b56ad385718f4fc5209 P 24c8c38e712041bca6609c1d45165595 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d658996110e4a3c83f948c468938fd3 in term 1.
I20260619 16:43:16.855777 1740 leader_election.cc:302] T 64d666c55f4a4b56ad385718f4fc5209 P 3d658996110e4a3c83f948c468938fd3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d658996110e4a3c83f948c468938fd3, 560a8f774fe846b4b173a721079b3830; no voters:
I20260619 16:43:16.856069 2021 raft_consensus.cc:2781] T 64d666c55f4a4b56ad385718f4fc5209 P 3d658996110e4a3c83f948c468938fd3 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:43:16.857024 2021 raft_consensus.cc:686] T 64d666c55f4a4b56ad385718f4fc5209 P 3d658996110e4a3c83f948c468938fd3 [term 1 LEADER]: Becoming Leader. State: Replica: 3d658996110e4a3c83f948c468938fd3, State: Running, Role: LEADER
I20260619 16:43:16.857491 2021 consensus_queue.cc:237] T 64d666c55f4a4b56ad385718f4fc5209 P 3d658996110e4a3c83f948c468938fd3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d658996110e4a3c83f948c468938fd3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44995 } } peers { permanent_uuid: "560a8f774fe846b4b173a721079b3830" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38031 } } peers { permanent_uuid: "24c8c38e712041bca6609c1d45165595" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42937 } }
I20260619 16:43:16.861826 1511 catalog_manager.cc:5495] T 64d666c55f4a4b56ad385718f4fc5209 P 3d658996110e4a3c83f948c468938fd3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3d658996110e4a3c83f948c468938fd3 (127.0.97.2). New cstate: current_term: 1 leader_uuid: "3d658996110e4a3c83f948c468938fd3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d658996110e4a3c83f948c468938fd3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44995 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "560a8f774fe846b4b173a721079b3830" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38031 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "24c8c38e712041bca6609c1d45165595" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42937 } health_report { overall_health: UNKNOWN } } }
W20260619 16:43:16.878854 1858 tablet.cc:2369] T 64d666c55f4a4b56ad385718f4fc5209 P 3d658996110e4a3c83f948c468938fd3: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 16:43:16.915618 1719 tablet.cc:2369] T 64d666c55f4a4b56ad385718f4fc5209 P 560a8f774fe846b4b173a721079b3830: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 16:43:16.919404 2011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:16.914261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49562 (local address 127.0.97.2:44995)
0619 16:43:16.914449 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0619 16:43:16.914456 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:43:16.914474 (+ 18us) server_negotiation.cc:407] Connection header received
0619 16:43:16.914779 (+ 305us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:43:16.914783 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:43:16.914865 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:43:16.914990 (+ 125us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 16:43:16.915348 (+ 358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:16.916097 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:16.917387 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:16.917809 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:16.917912 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:43:16.918369 (+ 457us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 16:43:16.918852 (+ 483us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 16:43:16.918971 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0619 16:43:16.919085 (+ 114us) server_negotiation.cc:299] Negotiation successful
0619 16:43:16.919160 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260619 16:43:16.927500 1668 raft_consensus.cc:1261] T 64d666c55f4a4b56ad385718f4fc5209 P 560a8f774fe846b4b173a721079b3830 [term 1 FOLLOWER]: Refusing update from remote peer 3d658996110e4a3c83f948c468938fd3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:43:16.927500 1951 raft_consensus.cc:1261] T 64d666c55f4a4b56ad385718f4fc5209 P 24c8c38e712041bca6609c1d45165595 [term 1 FOLLOWER]: Refusing update from remote peer 3d658996110e4a3c83f948c468938fd3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:43:16.928511 2026 consensus_queue.cc:1035] T 64d666c55f4a4b56ad385718f4fc5209 P 3d658996110e4a3c83f948c468938fd3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "24c8c38e712041bca6609c1d45165595" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:43:16.928826 2021 consensus_queue.cc:1035] T 64d666c55f4a4b56ad385718f4fc5209 P 3d658996110e4a3c83f948c468938fd3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "560a8f774fe846b4b173a721079b3830" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38031 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:43:16.941696 2030 mvcc.cc:205] Tried to move back new op lower bound from 7298610777805467648 to 7298610777532239872. Current Snapshot: MvccSnapshot[applied={T|T < 7298610777805467648}]
I20260619 16:43:16.957002 1511 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40224:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:43:16.957247 1511 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40224:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:43:16.960547 1511 catalog_manager.cc:5782] T 00000000000000000000000000000000 P de31151854fc4a318c6bc1f46f6b59bf: Sending DeleteTablet for 3 replicas of tablet 64d666c55f4a4b56ad385718f4fc5209
I20260619 16:43:16.961629 1787 tablet_service.cc:1514] Processing DeleteTablet for tablet 64d666c55f4a4b56ad385718f4fc5209 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:43:16 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:49556
I20260619 16:43:16.961879 1926 tablet_service.cc:1514] Processing DeleteTablet for tablet 64d666c55f4a4b56ad385718f4fc5209 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:43:16 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:57588
I20260619 16:43:16.962502 1648 tablet_service.cc:1514] Processing DeleteTablet for tablet 64d666c55f4a4b56ad385718f4fc5209 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:43:16 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:60058
I20260619 16:43:16.963476 1511 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40224:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 16:43:16.964210 1511 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 16:43:16.966861 2041 tablet_replica.cc:330] T 64d666c55f4a4b56ad385718f4fc5209 P 560a8f774fe846b4b173a721079b3830: stopping tablet replica
I20260619 16:43:16.967361 2040 tablet_replica.cc:330] T 64d666c55f4a4b56ad385718f4fc5209 P 24c8c38e712041bca6609c1d45165595: stopping tablet replica
I20260619 16:43:16.967824 2040 raft_consensus.cc:2229] T 64d666c55f4a4b56ad385718f4fc5209 P 24c8c38e712041bca6609c1d45165595 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 16:43:16.968446 2041 raft_consensus.cc:2229] T 64d666c55f4a4b56ad385718f4fc5209 P 560a8f774fe846b4b173a721079b3830 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 16:43:16.969020 2040 raft_consensus.cc:2258] T 64d666c55f4a4b56ad385718f4fc5209 P 24c8c38e712041bca6609c1d45165595 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:43:16.969547 2041 raft_consensus.cc:2258] T 64d666c55f4a4b56ad385718f4fc5209 P 560a8f774fe846b4b173a721079b3830 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:43:16.972045 2040 ts_tablet_manager.cc:1905] T 64d666c55f4a4b56ad385718f4fc5209 P 24c8c38e712041bca6609c1d45165595: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:43:16.972203 2039 tablet_replica.cc:330] T 64d666c55f4a4b56ad385718f4fc5209 P 3d658996110e4a3c83f948c468938fd3: stopping tablet replica
I20260619 16:43:16.972656 2039 raft_consensus.cc:2229] T 64d666c55f4a4b56ad385718f4fc5209 P 3d658996110e4a3c83f948c468938fd3 [term 1 LEADER]: Raft consensus shutting down.
I20260619 16:43:16.972779 1786 tablet_service.cc:1467] Processing CreateTablet for tablet 8d32effa13d0408daf62005fb5f0068b (DEFAULT_TABLE table=test-table [id=37549e254e014be88ec11191c82e6fdf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:43:16.973354 2039 raft_consensus.cc:2258] T 64d666c55f4a4b56ad385718f4fc5209 P 3d658996110e4a3c83f948c468938fd3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:43:16.974208 1648 tablet_service.cc:1467] Processing CreateTablet for tablet 8d32effa13d0408daf62005fb5f0068b (DEFAULT_TABLE table=test-table [id=37549e254e014be88ec11191c82e6fdf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:43:16.974594 1648 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d32effa13d0408daf62005fb5f0068b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:43:16.975366 2039 ts_tablet_manager.cc:1905] T 64d666c55f4a4b56ad385718f4fc5209 P 3d658996110e4a3c83f948c468938fd3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:43:16.973351 1786 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d32effa13d0408daf62005fb5f0068b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:43:16.978144 1926 tablet_service.cc:1467] Processing CreateTablet for tablet 8d32effa13d0408daf62005fb5f0068b (DEFAULT_TABLE table=test-table [id=37549e254e014be88ec11191c82e6fdf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:43:16.978518 1926 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d32effa13d0408daf62005fb5f0068b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:43:16.979455 2040 ts_tablet_manager.cc:1918] T 64d666c55f4a4b56ad385718f4fc5209 P 24c8c38e712041bca6609c1d45165595: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 16:43:16.979604 2040 log.cc:1196] T 64d666c55f4a4b56ad385718f4fc5209 P 24c8c38e712041bca6609c1d45165595: Deleting WAL directory at /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887385903558-388-0/minicluster-data/ts-2/wal/wals/64d666c55f4a4b56ad385718f4fc5209
I20260619 16:43:16.980099 2040 ts_tablet_manager.cc:1939] T 64d666c55f4a4b56ad385718f4fc5209 P 24c8c38e712041bca6609c1d45165595: Deleting consensus metadata
I20260619 16:43:16.981834 1497 catalog_manager.cc:4841] TS 24c8c38e712041bca6609c1d45165595 (127.0.97.3:42937): tablet 64d666c55f4a4b56ad385718f4fc5209 (table test-table [id=1b75f4cef8ab47b18cdd875fbaa3688b]) successfully deleted
I20260619 16:43:16.984673 2015 tablet_bootstrap.cc:492] T 8d32effa13d0408daf62005fb5f0068b P 3d658996110e4a3c83f948c468938fd3: Bootstrap starting.
I20260619 16:43:16.984913 2016 tablet_bootstrap.cc:492] T 8d32effa13d0408daf62005fb5f0068b P 560a8f774fe846b4b173a721079b3830: Bootstrap starting.
I20260619 16:43:16.985538 2017 tablet_bootstrap.cc:492] T 8d32effa13d0408daf62005fb5f0068b P 24c8c38e712041bca6609c1d45165595: Bootstrap starting.
I20260619 16:43:16.987048 2016 tablet_bootstrap.cc:654] T 8d32effa13d0408daf62005fb5f0068b P 560a8f774fe846b4b173a721079b3830: Neither blocks nor log segments found. Creating new log.
I20260619 16:43:16.987200 2017 tablet_bootstrap.cc:654] T 8d32effa13d0408daf62005fb5f0068b P 24c8c38e712041bca6609c1d45165595: Neither blocks nor log segments found. Creating new log.
I20260619 16:43:16.987172 2041 ts_tablet_manager.cc:1905] T 64d666c55f4a4b56ad385718f4fc5209 P 560a8f774fe846b4b173a721079b3830: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:43:16.990525 2015 tablet_bootstrap.cc:654] T 8d32effa13d0408daf62005fb5f0068b P 3d658996110e4a3c83f948c468938fd3: Neither blocks nor log segments found. Creating new log.
I20260619 16:43:16.997869 2017 tablet_bootstrap.cc:492] T 8d32effa13d0408daf62005fb5f0068b P 24c8c38e712041bca6609c1d45165595: No bootstrap required, opened a new log
I20260619 16:43:16.998104 2017 ts_tablet_manager.cc:1397] T 8d32effa13d0408daf62005fb5f0068b P 24c8c38e712041bca6609c1d45165595: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.007s
I20260619 16:43:16.999017 2017 raft_consensus.cc:348] T 8d32effa13d0408daf62005fb5f0068b P 24c8c38e712041bca6609c1d45165595 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "560a8f774fe846b4b173a721079b3830" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38031 } } peers { permanent_uuid: "3d658996110e4a3c83f948c468938fd3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44995 } } peers { permanent_uuid: "24c8c38e712041bca6609c1d45165595" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42937 } }
I20260619 16:43:16.999372 2017 raft_consensus.cc:374] T 8d32effa13d0408daf62005fb5f0068b P 24c8c38e712041bca6609c1d45165595 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:43:16.999583 2017 raft_consensus.cc:729] T 8d32effa13d0408daf62005fb5f0068b P 24c8c38e712041bca6609c1d45165595 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 24c8c38e712041bca6609c1d45165595, State: Initialized, Role: FOLLOWER
I20260619 16:43:17.000517 2017 consensus_queue.cc:260] T 8d32effa13d0408daf62005fb5f0068b P 24c8c38e712041bca6609c1d45165595 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "560a8f774fe846b4b173a721079b3830" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38031 } } peers { permanent_uuid: "3d658996110e4a3c83f948c468938fd3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44995 } } peers { permanent_uuid: "24c8c38e712041bca6609c1d45165595" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42937 } }
I20260619 16:43:17.002041 2017 ts_tablet_manager.cc:1428] T 8d32effa13d0408daf62005fb5f0068b P 24c8c38e712041bca6609c1d45165595: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260619 16:43:17.003631 2039 ts_tablet_manager.cc:1918] T 64d666c55f4a4b56ad385718f4fc5209 P 3d658996110e4a3c83f948c468938fd3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 16:43:17.003897 2016 tablet_bootstrap.cc:492] T 8d32effa13d0408daf62005fb5f0068b P 560a8f774fe846b4b173a721079b3830: No bootstrap required, opened a new log
I20260619 16:43:17.004137 2016 ts_tablet_manager.cc:1397] T 8d32effa13d0408daf62005fb5f0068b P 560a8f774fe846b4b173a721079b3830: Time spent bootstrapping tablet: real 0.019s user 0.004s sys 0.013s
I20260619 16:43:17.005162 2016 raft_consensus.cc:348] T 8d32effa13d0408daf62005fb5f0068b P 560a8f774fe846b4b173a721079b3830 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "560a8f774fe846b4b173a721079b3830" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38031 } } peers { permanent_uuid: "3d658996110e4a3c83f948c468938fd3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44995 } } peers { permanent_uuid: "24c8c38e712041bca6609c1d45165595" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42937 } }
I20260619 16:43:17.005390 2039 log.cc:1196] T 64d666c55f4a4b56ad385718f4fc5209 P 3d658996110e4a3c83f948c468938fd3: Deleting WAL directory at /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887385903558-388-0/minicluster-data/ts-1/wal/wals/64d666c55f4a4b56ad385718f4fc5209
I20260619 16:43:17.005460 2016 raft_consensus.cc:374] T 8d32effa13d0408daf62005fb5f0068b P 560a8f774fe846b4b173a721079b3830 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:43:17.005580 2016 raft_consensus.cc:729] T 8d32effa13d0408daf62005fb5f0068b P 560a8f774fe846b4b173a721079b3830 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 560a8f774fe846b4b173a721079b3830, State: Initialized, Role: FOLLOWER
I20260619 16:43:17.006062 2039 ts_tablet_manager.cc:1939] T 64d666c55f4a4b56ad385718f4fc5209 P 3d658996110e4a3c83f948c468938fd3: Deleting consensus metadata
I20260619 16:43:17.005964 2016 consensus_queue.cc:260] T 8d32effa13d0408daf62005fb5f0068b P 560a8f774fe846b4b173a721079b3830 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "560a8f774fe846b4b173a721079b3830" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38031 } } peers { permanent_uuid: "3d658996110e4a3c83f948c468938fd3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44995 } } peers { permanent_uuid: "24c8c38e712041bca6609c1d45165595" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42937 } }
I20260619 16:43:17.006515 2016 ts_tablet_manager.cc:1428] T 8d32effa13d0408daf62005fb5f0068b P 560a8f774fe846b4b173a721079b3830: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260619 16:43:17.007056 2041 ts_tablet_manager.cc:1918] T 64d666c55f4a4b56ad385718f4fc5209 P 560a8f774fe846b4b173a721079b3830: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 16:43:17.007269 2041 log.cc:1196] T 64d666c55f4a4b56ad385718f4fc5209 P 560a8f774fe846b4b173a721079b3830: Deleting WAL directory at /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887385903558-388-0/minicluster-data/ts-0/wal/wals/64d666c55f4a4b56ad385718f4fc5209
I20260619 16:43:17.007833 2041 ts_tablet_manager.cc:1939] T 64d666c55f4a4b56ad385718f4fc5209 P 560a8f774fe846b4b173a721079b3830: Deleting consensus metadata
I20260619 16:43:17.008692 1495 catalog_manager.cc:4841] TS 3d658996110e4a3c83f948c468938fd3 (127.0.97.2:44995): tablet 64d666c55f4a4b56ad385718f4fc5209 (table test-table [id=1b75f4cef8ab47b18cdd875fbaa3688b]) successfully deleted
I20260619 16:43:17.009683 1497 catalog_manager.cc:4841] TS 560a8f774fe846b4b173a721079b3830 (127.0.97.1:38031): tablet 64d666c55f4a4b56ad385718f4fc5209 (table test-table [id=1b75f4cef8ab47b18cdd875fbaa3688b]) successfully deleted
I20260619 16:43:17.011054 2015 tablet_bootstrap.cc:492] T 8d32effa13d0408daf62005fb5f0068b P 3d658996110e4a3c83f948c468938fd3: No bootstrap required, opened a new log
I20260619 16:43:17.011214 2015 ts_tablet_manager.cc:1397] T 8d32effa13d0408daf62005fb5f0068b P 3d658996110e4a3c83f948c468938fd3: Time spent bootstrapping tablet: real 0.027s user 0.003s sys 0.007s
I20260619 16:43:17.012035 2015 raft_consensus.cc:348] T 8d32effa13d0408daf62005fb5f0068b P 3d658996110e4a3c83f948c468938fd3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "560a8f774fe846b4b173a721079b3830" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38031 } } peers { permanent_uuid: "3d658996110e4a3c83f948c468938fd3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44995 } } peers { permanent_uuid: "24c8c38e712041bca6609c1d45165595" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42937 } }
I20260619 16:43:17.012274 2015 raft_consensus.cc:374] T 8d32effa13d0408daf62005fb5f0068b P 3d658996110e4a3c83f948c468938fd3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:43:17.012368 2015 raft_consensus.cc:729] T 8d32effa13d0408daf62005fb5f0068b P 3d658996110e4a3c83f948c468938fd3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d658996110e4a3c83f948c468938fd3, State: Initialized, Role: FOLLOWER
I20260619 16:43:17.012612 2015 consensus_queue.cc:260] T 8d32effa13d0408daf62005fb5f0068b P 3d658996110e4a3c83f948c468938fd3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "560a8f774fe846b4b173a721079b3830" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38031 } } peers { permanent_uuid: "3d658996110e4a3c83f948c468938fd3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44995 } } peers { permanent_uuid: "24c8c38e712041bca6609c1d45165595" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42937 } }
I20260619 16:43:17.013191 2015 ts_tablet_manager.cc:1428] T 8d32effa13d0408daf62005fb5f0068b P 3d658996110e4a3c83f948c468938fd3: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260619 16:43:17.208806 2022 raft_consensus.cc:482] T 8d32effa13d0408daf62005fb5f0068b P 24c8c38e712041bca6609c1d45165595 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 16:43:17.209034 2022 raft_consensus.cc:504] T 8d32effa13d0408daf62005fb5f0068b P 24c8c38e712041bca6609c1d45165595 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "560a8f774fe846b4b173a721079b3830" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38031 } } peers { permanent_uuid: "3d658996110e4a3c83f948c468938fd3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44995 } } peers { permanent_uuid: "24c8c38e712041bca6609c1d45165595" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42937 } }
I20260619 16:43:17.210475 2022 leader_election.cc:288] T 8d32effa13d0408daf62005fb5f0068b P 24c8c38e712041bca6609c1d45165595 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 560a8f774fe846b4b173a721079b3830 (127.0.97.1:38031), 3d658996110e4a3c83f948c468938fd3 (127.0.97.2:44995)
I20260619 16:43:17.213467 2021 raft_consensus.cc:482] T 8d32effa13d0408daf62005fb5f0068b P 3d658996110e4a3c83f948c468938fd3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 16:43:17.213635 2021 raft_consensus.cc:504] T 8d32effa13d0408daf62005fb5f0068b P 3d658996110e4a3c83f948c468938fd3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "560a8f774fe846b4b173a721079b3830" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38031 } } peers { permanent_uuid: "3d658996110e4a3c83f948c468938fd3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44995 } } peers { permanent_uuid: "24c8c38e712041bca6609c1d45165595" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42937 } }
I20260619 16:43:17.214287 2021 leader_election.cc:288] T 8d32effa13d0408daf62005fb5f0068b P 3d658996110e4a3c83f948c468938fd3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 560a8f774fe846b4b173a721079b3830 (127.0.97.1:38031), 24c8c38e712041bca6609c1d45165595 (127.0.97.3:42937)
I20260619 16:43:17.215104 1668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d32effa13d0408daf62005fb5f0068b" candidate_uuid: "3d658996110e4a3c83f948c468938fd3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "560a8f774fe846b4b173a721079b3830" is_pre_election: true
I20260619 16:43:17.215114 1951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d32effa13d0408daf62005fb5f0068b" candidate_uuid: "3d658996110e4a3c83f948c468938fd3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "24c8c38e712041bca6609c1d45165595" is_pre_election: true
I20260619 16:43:17.215354 1668 raft_consensus.cc:2454] T 8d32effa13d0408daf62005fb5f0068b P 560a8f774fe846b4b173a721079b3830 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d658996110e4a3c83f948c468938fd3 in term 0.
I20260619 16:43:17.215375 1951 raft_consensus.cc:2454] T 8d32effa13d0408daf62005fb5f0068b P 24c8c38e712041bca6609c1d45165595 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d658996110e4a3c83f948c468938fd3 in term 0.
I20260619 16:43:17.216254 1740 leader_election.cc:302] T 8d32effa13d0408daf62005fb5f0068b P 3d658996110e4a3c83f948c468938fd3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 24c8c38e712041bca6609c1d45165595, 3d658996110e4a3c83f948c468938fd3; no voters:
I20260619 16:43:17.216730 2021 raft_consensus.cc:2781] T 8d32effa13d0408daf62005fb5f0068b P 3d658996110e4a3c83f948c468938fd3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 16:43:17.216854 2021 raft_consensus.cc:482] T 8d32effa13d0408daf62005fb5f0068b P 3d658996110e4a3c83f948c468938fd3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 16:43:17.216943 2021 raft_consensus.cc:3037] T 8d32effa13d0408daf62005fb5f0068b P 3d658996110e4a3c83f948c468938fd3 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:43:17.218809 2011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:17.210904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:43023 (local address 127.0.97.2:44995)
0619 16:43:17.211121 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0619 16:43:17.211127 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:43:17.211614 (+ 487us) server_negotiation.cc:407] Connection header received
0619 16:43:17.211909 (+ 295us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:43:17.211914 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:43:17.211981 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:43:17.212061 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:43:17.212708 (+ 647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:17.213806 (+ 1098us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:17.216375 (+ 2569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:17.217647 (+ 1272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:17.217785 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:43:17.217885 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0619 16:43:17.218359 (+ 474us) server_negotiation.cc:299] Negotiation successful
0619 16:43:17.218501 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260619 16:43:17.219417 2046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:17.210776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:44995 (local address 127.0.97.3:43023)
0619 16:43:17.211507 (+ 731us) negotiation.cc:107] Waiting for socket to connect
0619 16:43:17.211527 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 16:43:17.211732 (+ 205us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:43:17.212267 (+ 535us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:43:17.212272 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:43:17.212297 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:43:17.212563 (+ 266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:43:17.212576 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:43:17.213957 (+ 1381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:43:17.213962 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:43:17.216180 (+ 2218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:43:17.216199 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:43:17.218010 (+ 1811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:43:17.218016 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:43:17.218186 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:43:17.218218 (+ 32us) client_negotiation.cc:769] Sending connection context
0619 16:43:17.218949 (+ 731us) client_negotiation.cc:240] Negotiation successful
0619 16:43:17.219085 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":624,"mutex_wait_us":240,"thread_start_us":919,"threads_started":1}
I20260619 16:43:17.219549 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:17.211262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:48693 (local address 127.0.97.1:38031)
0619 16:43:17.211431 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0619 16:43:17.211437 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:43:17.212058 (+ 621us) server_negotiation.cc:407] Connection header received
0619 16:43:17.212265 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:43:17.212270 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:43:17.212340 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:43:17.212425 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:43:17.213032 (+ 607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:17.214133 (+ 1101us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:17.217217 (+ 3084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:17.218438 (+ 1221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:17.218529 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:43:17.218621 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0619 16:43:17.219175 (+ 554us) server_negotiation.cc:299] Negotiation successful
0619 16:43:17.219299 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260619 16:43:17.220156 2045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:17.210790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:38031 (local address 127.0.97.3:48693)
0619 16:43:17.211956 (+ 1166us) negotiation.cc:107] Waiting for socket to connect
0619 16:43:17.211973 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 16:43:17.212104 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:43:17.212452 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:43:17.212457 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:43:17.212484 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:43:17.212807 (+ 323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:43:17.212819 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:43:17.214294 (+ 1475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:43:17.214300 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:43:17.217030 (+ 2730us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:43:17.217045 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:43:17.218793 (+ 1748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:43:17.218799 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:43:17.218985 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:43:17.219017 (+ 32us) client_negotiation.cc:769] Sending connection context
0619 16:43:17.219764 (+ 747us) client_negotiation.cc:240] Negotiation successful
0619 16:43:17.219893 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1050,"thread_start_us":633,"threads_started":1}
I20260619 16:43:17.220542 1807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d32effa13d0408daf62005fb5f0068b" candidate_uuid: "24c8c38e712041bca6609c1d45165595" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d658996110e4a3c83f948c468938fd3" is_pre_election: true
I20260619 16:43:17.220904 1668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d32effa13d0408daf62005fb5f0068b" candidate_uuid: "24c8c38e712041bca6609c1d45165595" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "560a8f774fe846b4b173a721079b3830" is_pre_election: true
I20260619 16:43:17.221163 1668 raft_consensus.cc:2454] T 8d32effa13d0408daf62005fb5f0068b P 560a8f774fe846b4b173a721079b3830 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 24c8c38e712041bca6609c1d45165595 in term 0.
I20260619 16:43:17.221329 2021 raft_consensus.cc:504] T 8d32effa13d0408daf62005fb5f0068b P 3d658996110e4a3c83f948c468938fd3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "560a8f774fe846b4b173a721079b3830" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38031 } } peers { permanent_uuid: "3d658996110e4a3c83f948c468938fd3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44995 } } peers { permanent_uuid: "24c8c38e712041bca6609c1d45165595" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42937 } }
I20260619 16:43:17.221813 1807 raft_consensus.cc:2379] T 8d32effa13d0408daf62005fb5f0068b P 3d658996110e4a3c83f948c468938fd3 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 24c8c38e712041bca6609c1d45165595 in current term 1: Already voted for candidate 3d658996110e4a3c83f948c468938fd3 in this term.
I20260619 16:43:17.221691 1879 leader_election.cc:302] T 8d32effa13d0408daf62005fb5f0068b P 24c8c38e712041bca6609c1d45165595 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 24c8c38e712041bca6609c1d45165595, 560a8f774fe846b4b173a721079b3830; no voters:
I20260619 16:43:17.222329 2022 raft_consensus.cc:2781] T 8d32effa13d0408daf62005fb5f0068b P 24c8c38e712041bca6609c1d45165595 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 16:43:17.222441 2022 raft_consensus.cc:482] T 8d32effa13d0408daf62005fb5f0068b P 24c8c38e712041bca6609c1d45165595 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 16:43:17.222535 2022 raft_consensus.cc:3037] T 8d32effa13d0408daf62005fb5f0068b P 24c8c38e712041bca6609c1d45165595 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:43:17.225069 2022 raft_consensus.cc:504] T 8d32effa13d0408daf62005fb5f0068b P 24c8c38e712041bca6609c1d45165595 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "560a8f774fe846b4b173a721079b3830" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38031 } } peers { permanent_uuid: "3d658996110e4a3c83f948c468938fd3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44995 } } peers { permanent_uuid: "24c8c38e712041bca6609c1d45165595" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42937 } }
I20260619 16:43:17.225626 2022 leader_election.cc:288] T 8d32effa13d0408daf62005fb5f0068b P 24c8c38e712041bca6609c1d45165595 [CANDIDATE]: Term 1 election: Requested vote from peers 560a8f774fe846b4b173a721079b3830 (127.0.97.1:38031), 3d658996110e4a3c83f948c468938fd3 (127.0.97.2:44995)
I20260619 16:43:17.226171 1807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d32effa13d0408daf62005fb5f0068b" candidate_uuid: "24c8c38e712041bca6609c1d45165595" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d658996110e4a3c83f948c468938fd3"
I20260619 16:43:17.226189 1668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d32effa13d0408daf62005fb5f0068b" candidate_uuid: "24c8c38e712041bca6609c1d45165595" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "560a8f774fe846b4b173a721079b3830"
I20260619 16:43:17.226291 2021 leader_election.cc:288] T 8d32effa13d0408daf62005fb5f0068b P 3d658996110e4a3c83f948c468938fd3 [CANDIDATE]: Term 1 election: Requested vote from peers 560a8f774fe846b4b173a721079b3830 (127.0.97.1:38031), 24c8c38e712041bca6609c1d45165595 (127.0.97.3:42937)
I20260619 16:43:17.226413 1807 raft_consensus.cc:2379] T 8d32effa13d0408daf62005fb5f0068b P 3d658996110e4a3c83f948c468938fd3 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 24c8c38e712041bca6609c1d45165595 in current term 1: Already voted for candidate 3d658996110e4a3c83f948c468938fd3 in this term.
I20260619 16:43:17.226393 1668 raft_consensus.cc:3037] T 8d32effa13d0408daf62005fb5f0068b P 560a8f774fe846b4b173a721079b3830 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:43:17.227042 1667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d32effa13d0408daf62005fb5f0068b" candidate_uuid: "3d658996110e4a3c83f948c468938fd3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "560a8f774fe846b4b173a721079b3830"
I20260619 16:43:17.227253 1951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d32effa13d0408daf62005fb5f0068b" candidate_uuid: "3d658996110e4a3c83f948c468938fd3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "24c8c38e712041bca6609c1d45165595"
I20260619 16:43:17.227484 1951 raft_consensus.cc:2379] T 8d32effa13d0408daf62005fb5f0068b P 24c8c38e712041bca6609c1d45165595 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 3d658996110e4a3c83f948c468938fd3 in current term 1: Already voted for candidate 24c8c38e712041bca6609c1d45165595 in this term.
I20260619 16:43:17.227913 1740 leader_election.cc:302] T 8d32effa13d0408daf62005fb5f0068b P 3d658996110e4a3c83f948c468938fd3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3d658996110e4a3c83f948c468938fd3; no voters: 24c8c38e712041bca6609c1d45165595, 560a8f774fe846b4b173a721079b3830
I20260619 16:43:17.228152 2021 raft_consensus.cc:2726] T 8d32effa13d0408daf62005fb5f0068b P 3d658996110e4a3c83f948c468938fd3 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260619 16:43:17.229403 1668 raft_consensus.cc:2454] T 8d32effa13d0408daf62005fb5f0068b P 560a8f774fe846b4b173a721079b3830 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 24c8c38e712041bca6609c1d45165595 in term 1.
I20260619 16:43:17.229768 1879 leader_election.cc:302] T 8d32effa13d0408daf62005fb5f0068b P 24c8c38e712041bca6609c1d45165595 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 24c8c38e712041bca6609c1d45165595, 560a8f774fe846b4b173a721079b3830; no voters: 3d658996110e4a3c83f948c468938fd3
I20260619 16:43:17.230041 2022 raft_consensus.cc:2781] T 8d32effa13d0408daf62005fb5f0068b P 24c8c38e712041bca6609c1d45165595 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:43:17.230430 2022 raft_consensus.cc:686] T 8d32effa13d0408daf62005fb5f0068b P 24c8c38e712041bca6609c1d45165595 [term 1 LEADER]: Becoming Leader. State: Replica: 24c8c38e712041bca6609c1d45165595, State: Running, Role: LEADER
I20260619 16:43:17.230851 2022 consensus_queue.cc:237] T 8d32effa13d0408daf62005fb5f0068b P 24c8c38e712041bca6609c1d45165595 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "560a8f774fe846b4b173a721079b3830" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38031 } } peers { permanent_uuid: "3d658996110e4a3c83f948c468938fd3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44995 } } peers { permanent_uuid: "24c8c38e712041bca6609c1d45165595" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42937 } }
I20260619 16:43:17.234194 1511 catalog_manager.cc:5495] T 8d32effa13d0408daf62005fb5f0068b P 24c8c38e712041bca6609c1d45165595 reported cstate change: term changed from 0 to 1, leader changed from <none> to 24c8c38e712041bca6609c1d45165595 (127.0.97.3). New cstate: current_term: 1 leader_uuid: "24c8c38e712041bca6609c1d45165595" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "560a8f774fe846b4b173a721079b3830" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38031 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3d658996110e4a3c83f948c468938fd3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44995 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "24c8c38e712041bca6609c1d45165595" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42937 } health_report { overall_health: HEALTHY } } }
Jun 19 16:43:17 dist-test-slave-wvz7 krb5kdc[1466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887394, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 16:43:17.318573 2055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:17.303957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34657 (local address 127.0.0.1:40236)
0619 16:43:17.304612 (+ 655us) negotiation.cc:107] Waiting for socket to connect
0619 16:43:17.304627 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 16:43:17.304759 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:43:17.305041 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:43:17.305045 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:43:17.305660 (+ 615us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:43:17.306307 (+ 647us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:43:17.306319 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:43:17.307344 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:43:17.307348 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:43:17.308667 (+ 1319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:43:17.308682 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:43:17.309428 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:43:17.309433 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:43:17.309599 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:43:17.310345 (+ 746us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:43:17.310367 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:43:17.312850 (+ 2483us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:43:17.316228 (+ 3378us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:43:17.316241 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:43:17.316249 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:43:17.316637 (+ 388us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:43:17.316950 (+ 313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:43:17.316955 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:43:17.316959 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:43:17.317077 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:43:17.317521 (+ 444us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:43:17.317529 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:43:17.317696 (+ 167us) client_negotiation.cc:769] Sending connection context
0619 16:43:17.317875 (+ 179us) client_negotiation.cc:240] Negotiation successful
0619 16:43:17.318156 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":537,"thread_start_us":412,"threads_started":1}
I20260619 16:43:17.318645 2054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:17.303911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40236 (local address 127.0.97.62:34657)
0619 16:43:17.304508 (+ 597us) server_negotiation.cc:206] Beginning negotiation
0619 16:43:17.304514 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:43:17.304702 (+ 188us) server_negotiation.cc:407] Connection header received
0619 16:43:17.304859 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:43:17.304863 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:43:17.304912 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:43:17.304994 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:43:17.306463 (+ 1469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:17.307214 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:17.308815 (+ 1601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:17.309279 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:17.309385 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:43:17.313040 (+ 3655us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:43:17.313045 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:43:17.313048 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:43:17.313077 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:43:17.316094 (+ 3017us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:43:17.316740 (+ 646us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:43:17.316745 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:43:17.316748 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:43:17.316858 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:43:17.317165 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:43:17.317170 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:43:17.317173 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:43:17.317408 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:43:17.317543 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0619 16:43:17.317988 (+ 445us) server_negotiation.cc:299] Negotiation successful
0619 16:43:17.318296 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":106,"server-negotiator.queue_time_us":449,"thread_start_us":296,"threads_started":1}
I20260619 16:43:17.324512 1511 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:40236:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260619 16:43:17.325489 1511 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 16:43:17.332098 1648 tablet_service.cc:1467] Processing CreateTablet for tablet cf7231f4ef084a3997eb6ff4496ec0f0 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=fe6cedf8b73c4b35bfd5c38a6918bd49]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 16:43:17.332472 1926 tablet_service.cc:1467] Processing CreateTablet for tablet cf7231f4ef084a3997eb6ff4496ec0f0 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=fe6cedf8b73c4b35bfd5c38a6918bd49]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 16:43:17.332646 1648 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cf7231f4ef084a3997eb6ff4496ec0f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:43:17.332947 1926 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cf7231f4ef084a3997eb6ff4496ec0f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:43:17.338498 1786 tablet_service.cc:1467] Processing CreateTablet for tablet cf7231f4ef084a3997eb6ff4496ec0f0 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=fe6cedf8b73c4b35bfd5c38a6918bd49]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 16:43:17.339051 1786 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cf7231f4ef084a3997eb6ff4496ec0f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:43:17.339515 2016 tablet_bootstrap.cc:492] T cf7231f4ef084a3997eb6ff4496ec0f0 P 560a8f774fe846b4b173a721079b3830: Bootstrap starting.
I20260619 16:43:17.339812 2017 tablet_bootstrap.cc:492] T cf7231f4ef084a3997eb6ff4496ec0f0 P 24c8c38e712041bca6609c1d45165595: Bootstrap starting.
I20260619 16:43:17.341117 2016 tablet_bootstrap.cc:654] T cf7231f4ef084a3997eb6ff4496ec0f0 P 560a8f774fe846b4b173a721079b3830: Neither blocks nor log segments found. Creating new log.
I20260619 16:43:17.341753 2017 tablet_bootstrap.cc:654] T cf7231f4ef084a3997eb6ff4496ec0f0 P 24c8c38e712041bca6609c1d45165595: Neither blocks nor log segments found. Creating new log.
I20260619 16:43:17.348769 2015 tablet_bootstrap.cc:492] T cf7231f4ef084a3997eb6ff4496ec0f0 P 3d658996110e4a3c83f948c468938fd3: Bootstrap starting.
I20260619 16:43:17.350514 2015 tablet_bootstrap.cc:654] T cf7231f4ef084a3997eb6ff4496ec0f0 P 3d658996110e4a3c83f948c468938fd3: Neither blocks nor log segments found. Creating new log.
I20260619 16:43:17.351856 2016 tablet_bootstrap.cc:492] T cf7231f4ef084a3997eb6ff4496ec0f0 P 560a8f774fe846b4b173a721079b3830: No bootstrap required, opened a new log
I20260619 16:43:17.352074 2016 ts_tablet_manager.cc:1397] T cf7231f4ef084a3997eb6ff4496ec0f0 P 560a8f774fe846b4b173a721079b3830: Time spent bootstrapping tablet: real 0.013s user 0.010s sys 0.001s
I20260619 16:43:17.352161 2017 tablet_bootstrap.cc:492] T cf7231f4ef084a3997eb6ff4496ec0f0 P 24c8c38e712041bca6609c1d45165595: No bootstrap required, opened a new log
I20260619 16:43:17.352308 2017 ts_tablet_manager.cc:1397] T cf7231f4ef084a3997eb6ff4496ec0f0 P 24c8c38e712041bca6609c1d45165595: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.005s
I20260619 16:43:17.353004 2017 raft_consensus.cc:348] T cf7231f4ef084a3997eb6ff4496ec0f0 P 24c8c38e712041bca6609c1d45165595 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "560a8f774fe846b4b173a721079b3830" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38031 } } peers { permanent_uuid: "3d658996110e4a3c83f948c468938fd3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44995 } } peers { permanent_uuid: "24c8c38e712041bca6609c1d45165595" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42937 } }
I20260619 16:43:17.353000 2016 raft_consensus.cc:348] T cf7231f4ef084a3997eb6ff4496ec0f0 P 560a8f774fe846b4b173a721079b3830 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "560a8f774fe846b4b173a721079b3830" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38031 } } peers { permanent_uuid: "3d658996110e4a3c83f948c468938fd3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44995 } } peers { permanent_uuid: "24c8c38e712041bca6609c1d45165595" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42937 } }
I20260619 16:43:17.353245 2017 raft_consensus.cc:374] T cf7231f4ef084a3997eb6ff4496ec0f0 P 24c8c38e712041bca6609c1d45165595 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:43:17.353317 2017 raft_consensus.cc:729] T cf7231f4ef084a3997eb6ff4496ec0f0 P 24c8c38e712041bca6609c1d45165595 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 24c8c38e712041bca6609c1d45165595, State: Initialized, Role: FOLLOWER
I20260619 16:43:17.353415 2016 raft_consensus.cc:374] T cf7231f4ef084a3997eb6ff4496ec0f0 P 560a8f774fe846b4b173a721079b3830 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:43:17.353530 2016 raft_consensus.cc:729] T cf7231f4ef084a3997eb6ff4496ec0f0 P 560a8f774fe846b4b173a721079b3830 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 560a8f774fe846b4b173a721079b3830, State: Initialized, Role: FOLLOWER
I20260619 16:43:17.353580 2017 consensus_queue.cc:260] T cf7231f4ef084a3997eb6ff4496ec0f0 P 24c8c38e712041bca6609c1d45165595 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "560a8f774fe846b4b173a721079b3830" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38031 } } peers { permanent_uuid: "3d658996110e4a3c83f948c468938fd3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44995 } } peers { permanent_uuid: "24c8c38e712041bca6609c1d45165595" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42937 } }
I20260619 16:43:17.353904 2016 consensus_queue.cc:260] T cf7231f4ef084a3997eb6ff4496ec0f0 P 560a8f774fe846b4b173a721079b3830 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "560a8f774fe846b4b173a721079b3830" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38031 } } peers { permanent_uuid: "3d658996110e4a3c83f948c468938fd3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44995 } } peers { permanent_uuid: "24c8c38e712041bca6609c1d45165595" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42937 } }
I20260619 16:43:17.353991 2022 tablet_replica.cc:439] T cf7231f4ef084a3997eb6ff4496ec0f0 P 24c8c38e712041bca6609c1d45165595: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "560a8f774fe846b4b173a721079b3830" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38031 } } peers { permanent_uuid: "3d658996110e4a3c83f948c468938fd3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44995 } } peers { permanent_uuid: "24c8c38e712041bca6609c1d45165595" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42937 } } }
I20260619 16:43:17.354229 2022 tablet_replica.cc:442] T cf7231f4ef084a3997eb6ff4496ec0f0 P 24c8c38e712041bca6609c1d45165595: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 16:43:17.354485 2023 tablet_replica.cc:439] T cf7231f4ef084a3997eb6ff4496ec0f0 P 560a8f774fe846b4b173a721079b3830: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "560a8f774fe846b4b173a721079b3830" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38031 } } peers { permanent_uuid: "3d658996110e4a3c83f948c468938fd3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44995 } } peers { permanent_uuid: "24c8c38e712041bca6609c1d45165595" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42937 } } }
I20260619 16:43:17.354785 2023 tablet_replica.cc:442] T cf7231f4ef084a3997eb6ff4496ec0f0 P 560a8f774fe846b4b173a721079b3830: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 16:43:17.355599 2017 ts_tablet_manager.cc:1428] T cf7231f4ef084a3997eb6ff4496ec0f0 P 24c8c38e712041bca6609c1d45165595: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260619 16:43:17.357317 2016 ts_tablet_manager.cc:1428] T cf7231f4ef084a3997eb6ff4496ec0f0 P 560a8f774fe846b4b173a721079b3830: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20260619 16:43:17.359114 2015 tablet_bootstrap.cc:492] T cf7231f4ef084a3997eb6ff4496ec0f0 P 3d658996110e4a3c83f948c468938fd3: No bootstrap required, opened a new log
I20260619 16:43:17.359282 2015 ts_tablet_manager.cc:1397] T cf7231f4ef084a3997eb6ff4496ec0f0 P 3d658996110e4a3c83f948c468938fd3: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.000s
I20260619 16:43:17.359962 2015 raft_consensus.cc:348] T cf7231f4ef084a3997eb6ff4496ec0f0 P 3d658996110e4a3c83f948c468938fd3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "560a8f774fe846b4b173a721079b3830" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38031 } } peers { permanent_uuid: "3d658996110e4a3c83f948c468938fd3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44995 } } peers { permanent_uuid: "24c8c38e712041bca6609c1d45165595" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42937 } }
I20260619 16:43:17.360203 2015 raft_consensus.cc:374] T cf7231f4ef084a3997eb6ff4496ec0f0 P 3d658996110e4a3c83f948c468938fd3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:43:17.360277 2015 raft_consensus.cc:729] T cf7231f4ef084a3997eb6ff4496ec0f0 P 3d658996110e4a3c83f948c468938fd3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d658996110e4a3c83f948c468938fd3, State: Initialized, Role: FOLLOWER
I20260619 16:43:17.360495 2015 consensus_queue.cc:260] T cf7231f4ef084a3997eb6ff4496ec0f0 P 3d658996110e4a3c83f948c468938fd3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "560a8f774fe846b4b173a721079b3830" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38031 } } peers { permanent_uuid: "3d658996110e4a3c83f948c468938fd3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44995 } } peers { permanent_uuid: "24c8c38e712041bca6609c1d45165595" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42937 } }
I20260619 16:43:17.360926 2021 tablet_replica.cc:439] T cf7231f4ef084a3997eb6ff4496ec0f0 P 3d658996110e4a3c83f948c468938fd3: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "560a8f774fe846b4b173a721079b3830" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38031 } } peers { permanent_uuid: "3d658996110e4a3c83f948c468938fd3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44995 } } peers { permanent_uuid: "24c8c38e712041bca6609c1d45165595" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42937 } } }
I20260619 16:43:17.361200 2021 tablet_replica.cc:442] T cf7231f4ef084a3997eb6ff4496ec0f0 P 3d658996110e4a3c83f948c468938fd3: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 16:43:17.361951 2015 ts_tablet_manager.cc:1428] T cf7231f4ef084a3997eb6ff4496ec0f0 P 3d658996110e4a3c83f948c468938fd3: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260619 16:43:17.440740 2023 raft_consensus.cc:482] T cf7231f4ef084a3997eb6ff4496ec0f0 P 560a8f774fe846b4b173a721079b3830 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 16:43:17.440995 2023 raft_consensus.cc:504] T cf7231f4ef084a3997eb6ff4496ec0f0 P 560a8f774fe846b4b173a721079b3830 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "560a8f774fe846b4b173a721079b3830" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38031 } } peers { permanent_uuid: "3d658996110e4a3c83f948c468938fd3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44995 } } peers { permanent_uuid: "24c8c38e712041bca6609c1d45165595" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42937 } }
I20260619 16:43:17.442667 2023 leader_election.cc:288] T cf7231f4ef084a3997eb6ff4496ec0f0 P 560a8f774fe846b4b173a721079b3830 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3d658996110e4a3c83f948c468938fd3 (127.0.97.2:44995), 24c8c38e712041bca6609c1d45165595 (127.0.97.3:42937)
I20260619 16:43:17.450160 2011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:17.443008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:50213 (local address 127.0.97.2:44995)
0619 16:43:17.443201 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0619 16:43:17.443207 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:43:17.443786 (+ 579us) server_negotiation.cc:407] Connection header received
0619 16:43:17.444089 (+ 303us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:43:17.444095 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:43:17.444171 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:43:17.444230 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:43:17.444951 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:17.445894 (+ 943us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:17.448281 (+ 2387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:17.449346 (+ 1065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:17.449478 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:43:17.449578 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0619 16:43:17.449832 (+ 254us) server_negotiation.cc:299] Negotiation successful
0619 16:43:17.449965 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260619 16:43:17.450340 2060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:17.443009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:44995 (local address 127.0.97.1:50213)
0619 16:43:17.443692 (+ 683us) negotiation.cc:107] Waiting for socket to connect
0619 16:43:17.443710 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 16:43:17.443903 (+ 193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:43:17.444399 (+ 496us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:43:17.444402 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:43:17.444423 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:43:17.444771 (+ 348us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:43:17.444784 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:43:17.446049 (+ 1265us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:43:17.446070 (+ 21us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:43:17.448118 (+ 2048us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:43:17.448132 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:43:17.449523 (+ 1391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:43:17.449528 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:43:17.449645 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:43:17.449669 (+ 24us) client_negotiation.cc:769] Sending connection context
0619 16:43:17.449811 (+ 142us) client_negotiation.cc:240] Negotiation successful
0619 16:43:17.449943 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":577,"mutex_wait_us":27,"thread_start_us":1044,"threads_started":1}
I20260619 16:43:17.450580 2061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:17.443622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:46329 (local address 127.0.97.3:42937)
0619 16:43:17.444099 (+ 477us) server_negotiation.cc:206] Beginning negotiation
0619 16:43:17.444104 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:43:17.444493 (+ 389us) server_negotiation.cc:407] Connection header received
0619 16:43:17.444652 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:43:17.444657 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:43:17.444726 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:43:17.444827 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:43:17.445339 (+ 512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:17.446337 (+ 998us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:17.448492 (+ 2155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:17.449657 (+ 1165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:17.449797 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:43:17.449890 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0619 16:43:17.450172 (+ 282us) server_negotiation.cc:299] Negotiation successful
0619 16:43:17.450300 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":371,"thread_start_us":234,"threads_started":1}
I20260619 16:43:17.450834 2059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:17.443059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:42937 (local address 127.0.97.1:46329)
0619 16:43:17.444395 (+ 1336us) negotiation.cc:107] Waiting for socket to connect
0619 16:43:17.444411 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 16:43:17.444531 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:43:17.444836 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:43:17.444839 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:43:17.444903 (+ 64us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:43:17.445198 (+ 295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:43:17.445206 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:43:17.446443 (+ 1237us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:43:17.446452 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:43:17.448354 (+ 1902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:43:17.448364 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:43:17.449870 (+ 1506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:43:17.449876 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:43:17.450020 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:43:17.450049 (+ 29us) client_negotiation.cc:769] Sending connection context
0619 16:43:17.450141 (+ 92us) client_negotiation.cc:240] Negotiation successful
0619 16:43:17.450264 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1171,"mutex_wait_us":1,"thread_start_us":374,"threads_started":1}
I20260619 16:43:17.451102 1807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf7231f4ef084a3997eb6ff4496ec0f0" candidate_uuid: "560a8f774fe846b4b173a721079b3830" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d658996110e4a3c83f948c468938fd3" is_pre_election: true
I20260619 16:43:17.451359 1807 raft_consensus.cc:2454] T cf7231f4ef084a3997eb6ff4496ec0f0 P 3d658996110e4a3c83f948c468938fd3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 560a8f774fe846b4b173a721079b3830 in term 0.
I20260619 16:43:17.451416 1951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf7231f4ef084a3997eb6ff4496ec0f0" candidate_uuid: "560a8f774fe846b4b173a721079b3830" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "24c8c38e712041bca6609c1d45165595" is_pre_election: true
I20260619 16:43:17.451659 1951 raft_consensus.cc:2454] T cf7231f4ef084a3997eb6ff4496ec0f0 P 24c8c38e712041bca6609c1d45165595 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 560a8f774fe846b4b173a721079b3830 in term 0.
I20260619 16:43:17.451864 1598 leader_election.cc:302] T cf7231f4ef084a3997eb6ff4496ec0f0 P 560a8f774fe846b4b173a721079b3830 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d658996110e4a3c83f948c468938fd3, 560a8f774fe846b4b173a721079b3830; no voters:
I20260619 16:43:17.452257 2023 raft_consensus.cc:2781] T cf7231f4ef084a3997eb6ff4496ec0f0 P 560a8f774fe846b4b173a721079b3830 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 16:43:17.452364 2023 raft_consensus.cc:482] T cf7231f4ef084a3997eb6ff4496ec0f0 P 560a8f774fe846b4b173a721079b3830 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 16:43:17.452453 2023 raft_consensus.cc:3037] T cf7231f4ef084a3997eb6ff4496ec0f0 P 560a8f774fe846b4b173a721079b3830 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:43:17.454913 2023 raft_consensus.cc:504] T cf7231f4ef084a3997eb6ff4496ec0f0 P 560a8f774fe846b4b173a721079b3830 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "560a8f774fe846b4b173a721079b3830" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38031 } } peers { permanent_uuid: "3d658996110e4a3c83f948c468938fd3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44995 } } peers { permanent_uuid: "24c8c38e712041bca6609c1d45165595" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42937 } }
I20260619 16:43:17.455524 2023 leader_election.cc:288] T cf7231f4ef084a3997eb6ff4496ec0f0 P 560a8f774fe846b4b173a721079b3830 [CANDIDATE]: Term 1 election: Requested vote from peers 3d658996110e4a3c83f948c468938fd3 (127.0.97.2:44995), 24c8c38e712041bca6609c1d45165595 (127.0.97.3:42937)
I20260619 16:43:17.456036 1951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf7231f4ef084a3997eb6ff4496ec0f0" candidate_uuid: "560a8f774fe846b4b173a721079b3830" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "24c8c38e712041bca6609c1d45165595"
I20260619 16:43:17.456036 1807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf7231f4ef084a3997eb6ff4496ec0f0" candidate_uuid: "560a8f774fe846b4b173a721079b3830" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d658996110e4a3c83f948c468938fd3"
I20260619 16:43:17.456249 1807 raft_consensus.cc:3037] T cf7231f4ef084a3997eb6ff4496ec0f0 P 3d658996110e4a3c83f948c468938fd3 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:43:17.456247 1951 raft_consensus.cc:3037] T cf7231f4ef084a3997eb6ff4496ec0f0 P 24c8c38e712041bca6609c1d45165595 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:43:17.459010 1951 raft_consensus.cc:2454] T cf7231f4ef084a3997eb6ff4496ec0f0 P 24c8c38e712041bca6609c1d45165595 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 560a8f774fe846b4b173a721079b3830 in term 1.
I20260619 16:43:17.459010 1807 raft_consensus.cc:2454] T cf7231f4ef084a3997eb6ff4496ec0f0 P 3d658996110e4a3c83f948c468938fd3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 560a8f774fe846b4b173a721079b3830 in term 1.
I20260619 16:43:17.459426 1600 leader_election.cc:302] T cf7231f4ef084a3997eb6ff4496ec0f0 P 560a8f774fe846b4b173a721079b3830 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 24c8c38e712041bca6609c1d45165595, 560a8f774fe846b4b173a721079b3830; no voters:
I20260619 16:43:17.459766 2023 raft_consensus.cc:2781] T cf7231f4ef084a3997eb6ff4496ec0f0 P 560a8f774fe846b4b173a721079b3830 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:43:17.460131 2023 raft_consensus.cc:686] T cf7231f4ef084a3997eb6ff4496ec0f0 P 560a8f774fe846b4b173a721079b3830 [term 1 LEADER]: Becoming Leader. State: Replica: 560a8f774fe846b4b173a721079b3830, State: Running, Role: LEADER
I20260619 16:43:17.460443 2023 consensus_queue.cc:237] T cf7231f4ef084a3997eb6ff4496ec0f0 P 560a8f774fe846b4b173a721079b3830 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "560a8f774fe846b4b173a721079b3830" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38031 } } peers { permanent_uuid: "3d658996110e4a3c83f948c468938fd3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44995 } } peers { permanent_uuid: "24c8c38e712041bca6609c1d45165595" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42937 } }
I20260619 16:43:17.462299 2062 tablet_replica.cc:439] T cf7231f4ef084a3997eb6ff4496ec0f0 P 560a8f774fe846b4b173a721079b3830: TxnStatusTablet state changed. Reason: New leader 560a8f774fe846b4b173a721079b3830. Latest consensus state: current_term: 1 leader_uuid: "560a8f774fe846b4b173a721079b3830" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "560a8f774fe846b4b173a721079b3830" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38031 } } peers { permanent_uuid: "3d658996110e4a3c83f948c468938fd3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44995 } } peers { permanent_uuid: "24c8c38e712041bca6609c1d45165595" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42937 } } }
I20260619 16:43:17.462718 2062 tablet_replica.cc:442] T cf7231f4ef084a3997eb6ff4496ec0f0 P 560a8f774fe846b4b173a721079b3830: This TxnStatusTablet replica's current role is: LEADER
I20260619 16:43:17.463307 2064 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260619 16:43:17.463496 2064 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260619 16:43:17.463758 1509 catalog_manager.cc:5495] T cf7231f4ef084a3997eb6ff4496ec0f0 P 560a8f774fe846b4b173a721079b3830 reported cstate change: term changed from 0 to 1, leader changed from <none> to 560a8f774fe846b4b173a721079b3830 (127.0.97.1). New cstate: current_term: 1 leader_uuid: "560a8f774fe846b4b173a721079b3830" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "560a8f774fe846b4b173a721079b3830" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38031 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3d658996110e4a3c83f948c468938fd3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44995 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "24c8c38e712041bca6609c1d45165595" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42937 } health_report { overall_health: UNKNOWN } } }
I20260619 16:43:17.509852 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:17.503834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60060 (local address 127.0.97.1:38031)
0619 16:43:17.504026 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0619 16:43:17.504032 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:43:17.504106 (+ 74us) server_negotiation.cc:407] Connection header received
0619 16:43:17.504326 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:43:17.504332 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:43:17.504435 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:43:17.504594 (+ 159us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 16:43:17.504892 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:17.506109 (+ 1217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:17.507367 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:17.508003 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:17.508131 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:43:17.508498 (+ 367us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 16:43:17.509234 (+ 736us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 16:43:17.509369 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0619 16:43:17.509458 (+ 89us) server_negotiation.cc:299] Negotiation successful
0619 16:43:17.509537 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260619 16:43:17.509902 2055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:17.503815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:38031 (local address 127.0.0.1:60060)
0619 16:43:17.504026 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0619 16:43:17.504045 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 16:43:17.504157 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:43:17.504619 (+ 462us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:43:17.504623 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:43:17.504632 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 16:43:17.504770 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:43:17.504776 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:43:17.506245 (+ 1469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:43:17.506249 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:43:17.507239 (+ 990us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:43:17.507251 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:43:17.508141 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:43:17.508145 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:43:17.508296 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:43:17.508348 (+ 52us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 16:43:17.509340 (+ 992us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 16:43:17.509357 (+ 17us) client_negotiation.cc:769] Sending connection context
0619 16:43:17.509426 (+ 69us) client_negotiation.cc:240] Negotiation successful
0619 16:43:17.509611 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45}
I20260619 16:43:17.515103 1951 raft_consensus.cc:1261] T cf7231f4ef084a3997eb6ff4496ec0f0 P 24c8c38e712041bca6609c1d45165595 [term 1 FOLLOWER]: Refusing update from remote peer 560a8f774fe846b4b173a721079b3830: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:43:17.515506 1807 raft_consensus.cc:1261] T cf7231f4ef084a3997eb6ff4496ec0f0 P 3d658996110e4a3c83f948c468938fd3 [term 1 FOLLOWER]: Refusing update from remote peer 560a8f774fe846b4b173a721079b3830: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:43:17.515877 2062 consensus_queue.cc:1035] T cf7231f4ef084a3997eb6ff4496ec0f0 P 560a8f774fe846b4b173a721079b3830 [LEADER]: Connected to new peer: Peer: permanent_uuid: "24c8c38e712041bca6609c1d45165595" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:43:17.516376 2023 consensus_queue.cc:1035] T cf7231f4ef084a3997eb6ff4496ec0f0 P 560a8f774fe846b4b173a721079b3830 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d658996110e4a3c83f948c468938fd3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44995 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:43:17.523254 2021 tablet_replica.cc:439] T cf7231f4ef084a3997eb6ff4496ec0f0 P 3d658996110e4a3c83f948c468938fd3: TxnStatusTablet state changed. Reason: New leader 560a8f774fe846b4b173a721079b3830. Latest consensus state: current_term: 1 leader_uuid: "560a8f774fe846b4b173a721079b3830" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "560a8f774fe846b4b173a721079b3830" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38031 } } peers { permanent_uuid: "3d658996110e4a3c83f948c468938fd3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44995 } } peers { permanent_uuid: "24c8c38e712041bca6609c1d45165595" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42937 } } }
I20260619 16:43:17.523576 2021 tablet_replica.cc:442] T cf7231f4ef084a3997eb6ff4496ec0f0 P 3d658996110e4a3c83f948c468938fd3: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 16:43:17.525352 2022 tablet_replica.cc:439] T cf7231f4ef084a3997eb6ff4496ec0f0 P 24c8c38e712041bca6609c1d45165595: TxnStatusTablet state changed. Reason: New leader 560a8f774fe846b4b173a721079b3830. Latest consensus state: current_term: 1 leader_uuid: "560a8f774fe846b4b173a721079b3830" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "560a8f774fe846b4b173a721079b3830" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38031 } } peers { permanent_uuid: "3d658996110e4a3c83f948c468938fd3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44995 } } peers { permanent_uuid: "24c8c38e712041bca6609c1d45165595" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42937 } } }
I20260619 16:43:17.525664 2022 tablet_replica.cc:442] T cf7231f4ef084a3997eb6ff4496ec0f0 P 24c8c38e712041bca6609c1d45165595: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 16:43:17.526546 2067 mvcc.cc:205] Tried to move back new op lower bound from 7298610780215275520 to 7298610780001157120. Current Snapshot: MvccSnapshot[applied={T|T < 7298610780215275520}]
I20260619 16:43:17.527472 2070 mvcc.cc:205] Tried to move back new op lower bound from 7298610780215275520 to 7298610780001157120. Current Snapshot: MvccSnapshot[applied={T|T < 7298610780215275520}]
I20260619 16:43:17.529861 2021 tablet_replica.cc:439] T cf7231f4ef084a3997eb6ff4496ec0f0 P 3d658996110e4a3c83f948c468938fd3: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "560a8f774fe846b4b173a721079b3830" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "560a8f774fe846b4b173a721079b3830" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38031 } } peers { permanent_uuid: "3d658996110e4a3c83f948c468938fd3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44995 } } peers { permanent_uuid: "24c8c38e712041bca6609c1d45165595" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42937 } } }
I20260619 16:43:17.530102 2021 tablet_replica.cc:442] T cf7231f4ef084a3997eb6ff4496ec0f0 P 3d658996110e4a3c83f948c468938fd3: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 16:43:17.530359 2022 tablet_replica.cc:439] T cf7231f4ef084a3997eb6ff4496ec0f0 P 24c8c38e712041bca6609c1d45165595: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "560a8f774fe846b4b173a721079b3830" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "560a8f774fe846b4b173a721079b3830" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38031 } } peers { permanent_uuid: "3d658996110e4a3c83f948c468938fd3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44995 } } peers { permanent_uuid: "24c8c38e712041bca6609c1d45165595" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42937 } } }
I20260619 16:43:17.530649 2022 tablet_replica.cc:442] T cf7231f4ef084a3997eb6ff4496ec0f0 P 24c8c38e712041bca6609c1d45165595: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 16:43:17.532260 2023 tablet_replica.cc:439] T cf7231f4ef084a3997eb6ff4496ec0f0 P 560a8f774fe846b4b173a721079b3830: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "560a8f774fe846b4b173a721079b3830" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "560a8f774fe846b4b173a721079b3830" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38031 } } peers { permanent_uuid: "3d658996110e4a3c83f948c468938fd3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44995 } } peers { permanent_uuid: "24c8c38e712041bca6609c1d45165595" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42937 } } }
I20260619 16:43:17.532521 2023 tablet_replica.cc:442] T cf7231f4ef084a3997eb6ff4496ec0f0 P 560a8f774fe846b4b173a721079b3830: This TxnStatusTablet replica's current role is: LEADER
I20260619 16:43:17.532766 2023 tablet_replica.cc:439] T cf7231f4ef084a3997eb6ff4496ec0f0 P 560a8f774fe846b4b173a721079b3830: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "560a8f774fe846b4b173a721079b3830" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "560a8f774fe846b4b173a721079b3830" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38031 } } peers { permanent_uuid: "3d658996110e4a3c83f948c468938fd3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44995 } } peers { permanent_uuid: "24c8c38e712041bca6609c1d45165595" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42937 } } }
I20260619 16:43:17.532956 2023 tablet_replica.cc:442] T cf7231f4ef084a3997eb6ff4496ec0f0 P 560a8f774fe846b4b173a721079b3830: This TxnStatusTablet replica's current role is: LEADER
I20260619 16:43:17.549796 2061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:17.544238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57590 (local address 127.0.97.3:42937)
0619 16:43:17.544405 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0619 16:43:17.544411 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:43:17.544855 (+ 444us) server_negotiation.cc:407] Connection header received
0619 16:43:17.545056 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:43:17.545060 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:43:17.545151 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:43:17.545288 (+ 137us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 16:43:17.545723 (+ 435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:17.546496 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:17.547795 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:17.548212 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:17.548316 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:43:17.548698 (+ 382us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 16:43:17.549211 (+ 513us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 16:43:17.549344 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0619 16:43:17.549429 (+ 85us) server_negotiation.cc:299] Negotiation successful
0619 16:43:17.549504 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260619 16:43:17.564994 2080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:17.559207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:38031 (local address 127.0.97.3:59905)
0619 16:43:17.559755 (+ 548us) negotiation.cc:107] Waiting for socket to connect
0619 16:43:17.559770 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 16:43:17.559887 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:43:17.560185 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:43:17.560187 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:43:17.560196 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 16:43:17.560444 (+ 248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:43:17.560451 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:43:17.561883 (+ 1432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:43:17.561887 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:43:17.563018 (+ 1131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:43:17.563030 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:43:17.563996 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:43:17.564001 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:43:17.564109 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:43:17.564166 (+ 57us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 16:43:17.564527 (+ 361us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 16:43:17.564533 (+ 6us) client_negotiation.cc:769] Sending connection context
0619 16:43:17.564582 (+ 49us) client_negotiation.cc:240] Negotiation successful
0619 16:43:17.564695 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":392,"thread_start_us":275,"threads_started":1}
I20260619 16:43:17.565012 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:17.559054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:59905 (local address 127.0.97.1:38031)
0619 16:43:17.559221 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0619 16:43:17.559235 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 16:43:17.559849 (+ 614us) server_negotiation.cc:407] Connection header received
0619 16:43:17.559992 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:43:17.559998 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:43:17.560078 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:43:17.560174 (+ 96us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 16:43:17.560571 (+ 397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:17.561770 (+ 1199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:17.563142 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:17.563855 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:17.563976 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:43:17.564318 (+ 342us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 16:43:17.564447 (+ 129us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 16:43:17.564531 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0619 16:43:17.564612 (+ 81us) server_negotiation.cc:299] Negotiation successful
0619 16:43:17.564692 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260619 16:43:17.574752 1807 raft_consensus.cc:1261] T 8d32effa13d0408daf62005fb5f0068b P 3d658996110e4a3c83f948c468938fd3 [term 1 FOLLOWER]: Refusing update from remote peer 24c8c38e712041bca6609c1d45165595: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:43:17.575124 1668 raft_consensus.cc:1261] T 8d32effa13d0408daf62005fb5f0068b P 560a8f774fe846b4b173a721079b3830 [term 1 FOLLOWER]: Refusing update from remote peer 24c8c38e712041bca6609c1d45165595: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:43:17.575385 2047 consensus_queue.cc:1035] T 8d32effa13d0408daf62005fb5f0068b P 24c8c38e712041bca6609c1d45165595 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d658996110e4a3c83f948c468938fd3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44995 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:43:17.575742 2022 consensus_queue.cc:1035] T 8d32effa13d0408daf62005fb5f0068b P 24c8c38e712041bca6609c1d45165595 [LEADER]: Connected to new peer: Peer: permanent_uuid: "560a8f774fe846b4b173a721079b3830" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38031 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:43:17.626883 2061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:17.619196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:34363 (local address 127.0.97.3:42937)
0619 16:43:17.619361 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0619 16:43:17.619367 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:43:17.619528 (+ 161us) server_negotiation.cc:407] Connection header received
0619 16:43:17.619806 (+ 278us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:43:17.619812 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:43:17.619891 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:43:17.619948 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 16:43:17.620716 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:17.621953 (+ 1237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:17.624320 (+ 2367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:17.625002 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:17.625096 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:43:17.626108 (+ 1012us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 16:43:17.626251 (+ 143us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 16:43:17.626304 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0619 16:43:17.626504 (+ 200us) server_negotiation.cc:299] Negotiation successful
0619 16:43:17.626599 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260619 16:43:17.627017 2085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:17.618776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:42937 (local address 127.0.97.1:34363)
0619 16:43:17.619426 (+ 650us) negotiation.cc:107] Waiting for socket to connect
0619 16:43:17.619446 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 16:43:17.619653 (+ 207us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:43:17.620132 (+ 479us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:43:17.620137 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:43:17.620149 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 16:43:17.620550 (+ 401us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:43:17.620562 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:43:17.622092 (+ 1530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:43:17.622098 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:43:17.624140 (+ 2042us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:43:17.624170 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:43:17.625594 (+ 1424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:43:17.625600 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:43:17.625843 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:43:17.625912 (+ 69us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 16:43:17.626375 (+ 463us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 16:43:17.626384 (+ 9us) client_negotiation.cc:769] Sending connection context
0619 16:43:17.626456 (+ 72us) client_negotiation.cc:240] Negotiation successful
0619 16:43:17.626597 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":461,"thread_start_us":2234,"threads_started":1}
I20260619 16:43:17.668015 1511 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40224:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:43:17.668274 1511 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40224:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:43:17.671865 1511 catalog_manager.cc:5782] T 00000000000000000000000000000000 P de31151854fc4a318c6bc1f46f6b59bf: Sending DeleteTablet for 3 replicas of tablet 8d32effa13d0408daf62005fb5f0068b
I20260619 16:43:17.672983 1786 tablet_service.cc:1514] Processing DeleteTablet for tablet 8d32effa13d0408daf62005fb5f0068b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:43:17 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:49556
I20260619 16:43:17.673475 1926 tablet_service.cc:1514] Processing DeleteTablet for tablet 8d32effa13d0408daf62005fb5f0068b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:43:17 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:57588
I20260619 16:43:17.674010 1648 tablet_service.cc:1514] Processing DeleteTablet for tablet 8d32effa13d0408daf62005fb5f0068b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:43:17 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:60058
I20260619 16:43:17.692355 2089 tablet_replica.cc:330] T 8d32effa13d0408daf62005fb5f0068b P 3d658996110e4a3c83f948c468938fd3: stopping tablet replica
I20260619 16:43:17.693023 2089 raft_consensus.cc:2229] T 8d32effa13d0408daf62005fb5f0068b P 3d658996110e4a3c83f948c468938fd3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 16:43:17.693312 2089 raft_consensus.cc:2258] T 8d32effa13d0408daf62005fb5f0068b P 3d658996110e4a3c83f948c468938fd3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:43:17.697294 2089 ts_tablet_manager.cc:1905] T 8d32effa13d0408daf62005fb5f0068b P 3d658996110e4a3c83f948c468938fd3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:43:17.708138 2091 tablet_replica.cc:330] T 8d32effa13d0408daf62005fb5f0068b P 560a8f774fe846b4b173a721079b3830: stopping tablet replica
I20260619 16:43:17.709266 2090 tablet_replica.cc:330] T 8d32effa13d0408daf62005fb5f0068b P 24c8c38e712041bca6609c1d45165595: stopping tablet replica
I20260619 16:43:17.709570 2091 raft_consensus.cc:2229] T 8d32effa13d0408daf62005fb5f0068b P 560a8f774fe846b4b173a721079b3830 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 16:43:17.710063 2091 raft_consensus.cc:2258] T 8d32effa13d0408daf62005fb5f0068b P 560a8f774fe846b4b173a721079b3830 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:43:17.710335 2090 raft_consensus.cc:2229] T 8d32effa13d0408daf62005fb5f0068b P 24c8c38e712041bca6609c1d45165595 [term 1 LEADER]: Raft consensus shutting down.
I20260619 16:43:17.711088 2090 raft_consensus.cc:2258] T 8d32effa13d0408daf62005fb5f0068b P 24c8c38e712041bca6609c1d45165595 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:43:17.712288 2091 ts_tablet_manager.cc:1905] T 8d32effa13d0408daf62005fb5f0068b P 560a8f774fe846b4b173a721079b3830: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:43:17.712620 2090 ts_tablet_manager.cc:1905] T 8d32effa13d0408daf62005fb5f0068b P 24c8c38e712041bca6609c1d45165595: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:43:17.715368 2089 ts_tablet_manager.cc:1918] T 8d32effa13d0408daf62005fb5f0068b P 3d658996110e4a3c83f948c468938fd3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 16:43:17.715562 2089 log.cc:1196] T 8d32effa13d0408daf62005fb5f0068b P 3d658996110e4a3c83f948c468938fd3: Deleting WAL directory at /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887385903558-388-0/minicluster-data/ts-1/wal/wals/8d32effa13d0408daf62005fb5f0068b
I20260619 16:43:17.715969 2089 ts_tablet_manager.cc:1939] T 8d32effa13d0408daf62005fb5f0068b P 3d658996110e4a3c83f948c468938fd3: Deleting consensus metadata
I20260619 16:43:17.717005 1495 catalog_manager.cc:4841] TS 3d658996110e4a3c83f948c468938fd3 (127.0.97.2:44995): tablet 8d32effa13d0408daf62005fb5f0068b (table test-table [id=37549e254e014be88ec11191c82e6fdf]) successfully deleted
I20260619 16:43:17.725128 2091 ts_tablet_manager.cc:1918] T 8d32effa13d0408daf62005fb5f0068b P 560a8f774fe846b4b173a721079b3830: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 16:43:17.725265 2091 log.cc:1196] T 8d32effa13d0408daf62005fb5f0068b P 560a8f774fe846b4b173a721079b3830: Deleting WAL directory at /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887385903558-388-0/minicluster-data/ts-0/wal/wals/8d32effa13d0408daf62005fb5f0068b
I20260619 16:43:17.725689 2091 ts_tablet_manager.cc:1939] T 8d32effa13d0408daf62005fb5f0068b P 560a8f774fe846b4b173a721079b3830: Deleting consensus metadata
I20260619 16:43:17.726706 1497 catalog_manager.cc:4841] TS 560a8f774fe846b4b173a721079b3830 (127.0.97.1:38031): tablet 8d32effa13d0408daf62005fb5f0068b (table test-table [id=37549e254e014be88ec11191c82e6fdf]) successfully deleted
I20260619 16:43:17.728924 2090 ts_tablet_manager.cc:1918] T 8d32effa13d0408daf62005fb5f0068b P 24c8c38e712041bca6609c1d45165595: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 16:43:17.729058 2090 log.cc:1196] T 8d32effa13d0408daf62005fb5f0068b P 24c8c38e712041bca6609c1d45165595: Deleting WAL directory at /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887385903558-388-0/minicluster-data/ts-2/wal/wals/8d32effa13d0408daf62005fb5f0068b
I20260619 16:43:17.729486 2090 ts_tablet_manager.cc:1939] T 8d32effa13d0408daf62005fb5f0068b P 24c8c38e712041bca6609c1d45165595: Deleting consensus metadata
I20260619 16:43:17.730544 1497 catalog_manager.cc:4841] TS 24c8c38e712041bca6609c1d45165595 (127.0.97.3:42937): tablet 8d32effa13d0408daf62005fb5f0068b (table test-table [id=37549e254e014be88ec11191c82e6fdf]) successfully deleted
Jun 19 16:43:17 dist-test-slave-wvz7 krb5kdc[1466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887396, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260619 16:43:17.741032 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:17.707483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60068 (local address 127.0.97.1:38031)
0619 16:43:17.707663 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0619 16:43:17.707669 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:43:17.716482 (+ 8813us) server_negotiation.cc:407] Connection header received
0619 16:43:17.716762 (+ 280us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:43:17.716768 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:43:17.716851 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:43:17.716919 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:43:17.722542 (+ 5623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:17.723745 (+ 1203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:17.726948 (+ 3203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:17.727582 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:17.727692 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:43:17.733779 (+ 6087us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:43:17.733795 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:43:17.733814 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:43:17.733859 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:43:17.737672 (+ 3813us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:43:17.738519 (+ 847us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:43:17.738529 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:43:17.738533 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:43:17.738694 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:43:17.739133 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:43:17.739136 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:43:17.739138 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:43:17.739695 (+ 557us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:43:17.739985 (+ 290us) server_negotiation.cc:1035] Waiting for connection context
0619 16:43:17.740260 (+ 275us) server_negotiation.cc:299] Negotiation successful
0619 16:43:17.740485 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
W20260619 16:43:17.741791 1713 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60068
I20260619 16:43:17.760237 2054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:17.745223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40248 (local address 127.0.97.62:34657)
0619 16:43:17.745427 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0619 16:43:17.745433 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:43:17.745899 (+ 466us) server_negotiation.cc:407] Connection header received
0619 16:43:17.746081 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:43:17.746087 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:43:17.746168 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:43:17.746271 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:43:17.747764 (+ 1493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:17.748883 (+ 1119us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:17.750649 (+ 1766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:17.751271 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:17.751395 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:43:17.754171 (+ 2776us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:43:17.754178 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:43:17.754183 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:43:17.754223 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:43:17.757439 (+ 3216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:43:17.758236 (+ 797us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:43:17.758241 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:43:17.758245 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:43:17.758386 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:43:17.758765 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:43:17.758771 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:43:17.758773 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:43:17.759118 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:43:17.759347 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0619 16:43:17.759691 (+ 344us) server_negotiation.cc:299] Negotiation successful
0619 16:43:17.759910 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
W20260619 16:43:17.760890 1511 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40248
I20260619 16:43:17.763080 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu pid 1582
I20260619 16:43:17.781019 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:17.764201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60080 (local address 127.0.97.1:38031)
0619 16:43:17.764374 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0619 16:43:17.764383 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 16:43:17.766053 (+ 1670us) server_negotiation.cc:407] Connection header received
0619 16:43:17.766326 (+ 273us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:43:17.766331 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:43:17.766408 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:43:17.766517 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:43:17.767906 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:17.769124 (+ 1218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:17.770913 (+ 1789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:17.771677 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:17.771805 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:43:17.774607 (+ 2802us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:43:17.774613 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:43:17.774617 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:43:17.774655 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:43:17.778153 (+ 3498us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:43:17.778923 (+ 770us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:43:17.778929 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:43:17.778932 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:43:17.779103 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:43:17.779457 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:43:17.779462 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:43:17.779465 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:43:17.779793 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:43:17.780062 (+ 269us) server_negotiation.cc:1035] Waiting for connection context
0619 16:43:17.780357 (+ 295us) server_negotiation.cc:299] Negotiation successful
0619 16:43:17.780592 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
W20260619 16:43:17.781536 1713 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60080
W20260619 16:43:17.782217 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu pid 1582: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 16:43:17.782399 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu with pid 1582
I20260619 16:43:17.806514 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu pid 1722
Jun 19 16:43:17 dist-test-slave-wvz7 krb5kdc[1466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887396, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260619 16:43:17.822182 2011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:17.807260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49570 (local address 127.0.97.2:44995)
0619 16:43:17.807436 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0619 16:43:17.807442 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:43:17.807462 (+ 20us) server_negotiation.cc:407] Connection header received
0619 16:43:17.807581 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:43:17.807586 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:43:17.807664 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:43:17.807780 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:43:17.808769 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:17.809878 (+ 1109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:17.811036 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:17.811728 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:17.811849 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:43:17.815733 (+ 3884us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:43:17.815752 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:43:17.815772 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:43:17.815817 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:43:17.819351 (+ 3534us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:43:17.820154 (+ 803us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:43:17.820164 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:43:17.820168 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:43:17.820322 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:43:17.820638 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:43:17.820643 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:43:17.820646 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:43:17.821068 (+ 422us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:43:17.821408 (+ 340us) server_negotiation.cc:1035] Waiting for connection context
0619 16:43:17.821545 (+ 137us) server_negotiation.cc:299] Negotiation successful
0619 16:43:17.821800 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
W20260619 16:43:17.822820 1852 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49570
W20260619 16:43:17.823438 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu pid 1722: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 16:43:17.823611 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu with pid 1722
I20260619 16:43:17.848121 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu pid 1861
Jun 19 16:43:17 dist-test-slave-wvz7 krb5kdc[1466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887396, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260619 16:43:17.864362 2061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:17.848911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57604 (local address 127.0.97.3:42937)
0619 16:43:17.849056 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0619 16:43:17.849060 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:43:17.849077 (+ 17us) server_negotiation.cc:407] Connection header received
0619 16:43:17.849222 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:43:17.849225 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:43:17.849284 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:43:17.849369 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:43:17.850873 (+ 1504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:17.851644 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:17.853412 (+ 1768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:17.853963 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:17.854061 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:43:17.858869 (+ 4808us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:43:17.858882 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:43:17.858897 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:43:17.858932 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:43:17.861640 (+ 2708us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:43:17.862566 (+ 926us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:43:17.862573 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:43:17.862575 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:43:17.862702 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:43:17.863063 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:43:17.863066 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:43:17.863068 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:43:17.863449 (+ 381us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:43:17.863694 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0619 16:43:17.863948 (+ 254us) server_negotiation.cc:299] Negotiation successful
0619 16:43:17.864095 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
W20260619 16:43:17.864930 1991 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57604
W20260619 16:43:17.865494 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu pid 1861: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 16:43:17.865625 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu with pid 1861
I20260619 16:43:17.880012 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu pid 1479
I20260619 16:43:17.881016 1551 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:43:17.992440 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu with pid 1479
2026-06-19T16:43:18Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (4509 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887385903558-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 16:43:18 dist-test-slave-wvz7 krb5kdc[2100](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 16:43:18 dist-test-slave-wvz7 krb5kdc[2100](info): set up 2 sockets
Jun 19 16:43:18 dist-test-slave-wvz7 krb5kdc[2100](info): commencing operation
krb5kdc: starting...
W20260619 16:43:18.973572 388 mini_kdc.cc:121] Time spent starting KDC: real 0.944s user 0.001s sys 0.013s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887385903558-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887385903558-388-0/krb5kdc/test-user.keytab.
Jun 19 16:43:19 dist-test-slave-wvz7 krb5kdc[2100](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887399, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T16:43:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T16:43:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887385903558-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887385903558-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260619 16:43:19.139554 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu
/tmp/dist-test-taskEs2EH8/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887385903558-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887385903558-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887385903558-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887385903558-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:46341
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887385903558-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41637
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:46341
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887385903558-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887385903558-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887385903558-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:43:19.444116 2113 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:43:19.444437 2113 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:43:19.444567 2113 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:43:19.451700 2113 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:43:19.451817 2113 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260619 16:43:19.451886 2113 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:43:19.451970 2113 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:43:19.452046 2113 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:43:19.452129 2113 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:43:19.461879 2113 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41637
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887385903558-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887385903558-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:46341
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887385903558-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:46341
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887385903558-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887385903558-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:43:19.463142 2113 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:43:19.464391 2113 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:43:19.475764 2119 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:43:19.475723 2121 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:43:19.476310 2118 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:43:19.476421 2113 server_base.cc:1034] running on GCE node
I20260619 16:43:19.477227 2113 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:43:19.478685 2113 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:43:19.479949 2113 hybrid_clock.cc:648] HybridClock initialized: now 1781887399479862 us; error 37 us; skew 500 ppm
I20260619 16:43:19.480299 2113 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:43:19 dist-test-slave-wvz7 krb5kdc[2100](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887399, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:43:19.483367 2113 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260619 16:43:19.484761 2113 webserver.cc:460] Webserver started at http://127.0.97.62:45739/ using document root <none> and password file <none>
I20260619 16:43:19.485442 2113 fs_manager.cc:359] Metadata directory not provided
I20260619 16:43:19.485538 2113 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:43:19.485880 2113 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:43:19.488708 2113 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887385903558-388-0/minicluster-data/master-0/data/instance:
uuid: "58e545bbf9084b46bd00d2e25bf47bb3"
format_stamp: "Formatted at 2026-06-19 16:43:19 on dist-test-slave-wvz7"
server_key: "39907d6baea640a47a5e46ac5a9de706"
server_key_iv: "103274eed075375d2ff52d14bbb7a14e"
server_key_version: "encryptionkey@0"
I20260619 16:43:19.489440 2113 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887385903558-388-0/minicluster-data/master-0/wal/instance:
uuid: "58e545bbf9084b46bd00d2e25bf47bb3"
format_stamp: "Formatted at 2026-06-19 16:43:19 on dist-test-slave-wvz7"
server_key: "39907d6baea640a47a5e46ac5a9de706"
server_key_iv: "103274eed075375d2ff52d14bbb7a14e"
server_key_version: "encryptionkey@0"
I20260619 16:43:19.494544 2113 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260619 16:43:19.498044 2128 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:43:19.499636 2113 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260619 16:43:19.499815 2113 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887385903558-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887385903558-388-0/minicluster-data/master-0/wal
uuid: "58e545bbf9084b46bd00d2e25bf47bb3"
format_stamp: "Formatted at 2026-06-19 16:43:19 on dist-test-slave-wvz7"
server_key: "39907d6baea640a47a5e46ac5a9de706"
server_key_iv: "103274eed075375d2ff52d14bbb7a14e"
server_key_version: "encryptionkey@0"
I20260619 16:43:19.500038 2113 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887385903558-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887385903558-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887385903558-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:43:19.543155 2113 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:43:19.547637 2113 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:43:19.547899 2113 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:43:19.567545 2113 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:46341
I20260619 16:43:19.567593 2189 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:46341 every 8 connection(s)
I20260619 16:43:19.568800 2113 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887385903558-388-0/minicluster-data/master-0/data/info.pb
I20260619 16:43:19.572382 2190 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:43:19.575486 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu as pid 2113
I20260619 16:43:19.575673 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887385903558-388-0/minicluster-data/master-0/wal/instance
I20260619 16:43:19.576259 388 external_mini_cluster.cc:1428] Setting key 13ba5741848c6a8e50746c8670b7cd2c
I20260619 16:43:19.585120 2190 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 58e545bbf9084b46bd00d2e25bf47bb3: Bootstrap starting.
I20260619 16:43:19.588936 2190 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 58e545bbf9084b46bd00d2e25bf47bb3: Neither blocks nor log segments found. Creating new log.
I20260619 16:43:19.590006 2190 log.cc:824] T 00000000000000000000000000000000 P 58e545bbf9084b46bd00d2e25bf47bb3: Log is configured to *not* fsync() on all Append() calls
Jun 19 16:43:19 dist-test-slave-wvz7 krb5kdc[2100](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887399, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 16:43:19.592659 2190 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 58e545bbf9084b46bd00d2e25bf47bb3: No bootstrap required, opened a new log
I20260619 16:43:19.599313 2193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:19.578161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59512 (local address 127.0.97.62:46341)
0619 16:43:19.578930 (+ 769us) server_negotiation.cc:206] Beginning negotiation
0619 16:43:19.578946 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0619 16:43:19.578995 (+ 49us) server_negotiation.cc:407] Connection header received
0619 16:43:19.580765 (+ 1770us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:43:19.580793 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:43:19.581312 (+ 519us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:43:19.581797 (+ 485us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:43:19.583008 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:19.585112 (+ 2104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:19.586487 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:19.587584 (+ 1097us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:19.587780 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:43:19.591724 (+ 3944us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:43:19.591732 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:43:19.591749 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:43:19.591785 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:43:19.594957 (+ 3172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:43:19.595698 (+ 741us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:43:19.595705 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:43:19.595711 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:43:19.595915 (+ 204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:43:19.596309 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:43:19.596313 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:43:19.596315 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:43:19.596905 (+ 590us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:43:19.597218 (+ 313us) server_negotiation.cc:1035] Waiting for connection context
0619 16:43:19.597707 (+ 489us) server_negotiation.cc:299] Negotiation successful
0619 16:43:19.598178 (+ 471us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":523,"thread_start_us":318,"threads_started":1}
I20260619 16:43:19.600174 2190 raft_consensus.cc:348] T 00000000000000000000000000000000 P 58e545bbf9084b46bd00d2e25bf47bb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58e545bbf9084b46bd00d2e25bf47bb3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46341 } }
I20260619 16:43:19.600495 2190 raft_consensus.cc:374] T 00000000000000000000000000000000 P 58e545bbf9084b46bd00d2e25bf47bb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:43:19.600594 2190 raft_consensus.cc:729] T 00000000000000000000000000000000 P 58e545bbf9084b46bd00d2e25bf47bb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58e545bbf9084b46bd00d2e25bf47bb3, State: Initialized, Role: FOLLOWER
I20260619 16:43:19.601223 2190 consensus_queue.cc:260] T 00000000000000000000000000000000 P 58e545bbf9084b46bd00d2e25bf47bb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58e545bbf9084b46bd00d2e25bf47bb3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46341 } }
I20260619 16:43:19.601493 2190 raft_consensus.cc:388] T 00000000000000000000000000000000 P 58e545bbf9084b46bd00d2e25bf47bb3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:43:19.601624 2190 raft_consensus.cc:482] T 00000000000000000000000000000000 P 58e545bbf9084b46bd00d2e25bf47bb3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:43:19.601807 2190 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 58e545bbf9084b46bd00d2e25bf47bb3 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:43:19.603991 2190 raft_consensus.cc:504] T 00000000000000000000000000000000 P 58e545bbf9084b46bd00d2e25bf47bb3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58e545bbf9084b46bd00d2e25bf47bb3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46341 } }
I20260619 16:43:19.604456 2190 leader_election.cc:302] T 00000000000000000000000000000000 P 58e545bbf9084b46bd00d2e25bf47bb3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 58e545bbf9084b46bd00d2e25bf47bb3; no voters:
I20260619 16:43:19.604955 2190 leader_election.cc:288] T 00000000000000000000000000000000 P 58e545bbf9084b46bd00d2e25bf47bb3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 16:43:19.605065 2195 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 58e545bbf9084b46bd00d2e25bf47bb3 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:43:19.605692 2195 raft_consensus.cc:686] T 00000000000000000000000000000000 P 58e545bbf9084b46bd00d2e25bf47bb3 [term 1 LEADER]: Becoming Leader. State: Replica: 58e545bbf9084b46bd00d2e25bf47bb3, State: Running, Role: LEADER
I20260619 16:43:19.606177 2195 consensus_queue.cc:237] T 00000000000000000000000000000000 P 58e545bbf9084b46bd00d2e25bf47bb3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58e545bbf9084b46bd00d2e25bf47bb3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46341 } }
I20260619 16:43:19.606825 2190 sys_catalog.cc:564] T 00000000000000000000000000000000 P 58e545bbf9084b46bd00d2e25bf47bb3 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:43:19.608728 2197 sys_catalog.cc:455] T 00000000000000000000000000000000 P 58e545bbf9084b46bd00d2e25bf47bb3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 58e545bbf9084b46bd00d2e25bf47bb3. Latest consensus state: current_term: 1 leader_uuid: "58e545bbf9084b46bd00d2e25bf47bb3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58e545bbf9084b46bd00d2e25bf47bb3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46341 } } }
I20260619 16:43:19.608750 2196 sys_catalog.cc:455] T 00000000000000000000000000000000 P 58e545bbf9084b46bd00d2e25bf47bb3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "58e545bbf9084b46bd00d2e25bf47bb3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58e545bbf9084b46bd00d2e25bf47bb3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46341 } } }
I20260619 16:43:19.609186 2197 sys_catalog.cc:458] T 00000000000000000000000000000000 P 58e545bbf9084b46bd00d2e25bf47bb3 [sys.catalog]: This master's current role is: LEADER
I20260619 16:43:19.609223 2196 sys_catalog.cc:458] T 00000000000000000000000000000000 P 58e545bbf9084b46bd00d2e25bf47bb3 [sys.catalog]: This master's current role is: LEADER
I20260619 16:43:19.610172 2201 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:43:19.615633 2201 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:43:19.625259 2201 catalog_manager.cc:1347] Generated new cluster ID: fb9b08092f974da2a46ee72cb8509410
I20260619 16:43:19.625360 2201 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:43:19.633466 2201 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 16:43:19.635242 2201 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:43:19.641624 2201 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 58e545bbf9084b46bd00d2e25bf47bb3: Generated new TSK 0
I20260619 16:43:19.642612 2201 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887385903558-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887385903558-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260619 16:43:19.681859 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu
/tmp/dist-test-taskEs2EH8/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887385903558-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887385903558-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887385903558-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887385903558-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46341
--keytab_file=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887385903558-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41637
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887385903558-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887385903558-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887385903558-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:43:19.990000 2216 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:43:19.990281 2216 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:43:19.990404 2216 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:43:19.997750 2216 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:43:19.997880 2216 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:43:19.998004 2216 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260619 16:43:19.998070 2216 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 16:43:19.998172 2216 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260619 16:43:20.008456 2216 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41637
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887385903558-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887385903558-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887385903558-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887385903558-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:46341
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887385903558-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:43:20.009828 2216 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:43:20.011314 2216 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:43:20.024614 2221 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:43:20.024712 2222 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:43:20.025216 2224 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:43:20.025604 2216 server_base.cc:1034] running on GCE node
I20260619 16:43:20.026249 2216 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:43:20.027272 2216 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:43:20.028481 2216 hybrid_clock.cc:648] HybridClock initialized: now 1781887400028419 us; error 69 us; skew 500 ppm
I20260619 16:43:20.028846 2216 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:43:20 dist-test-slave-wvz7 krb5kdc[2100](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887400, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:43:20.035327 2216 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260619 16:43:20.036938 2216 webserver.cc:460] Webserver started at http://127.0.97.1:38407/ using document root <none> and password file <none>
I20260619 16:43:20.037698 2216 fs_manager.cc:359] Metadata directory not provided
I20260619 16:43:20.037858 2216 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:43:20.038157 2216 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:43:20.041064 2216 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887385903558-388-0/minicluster-data/ts-0/data/instance:
uuid: "0f3f5a7d5afd41ceb534f34d36810852"
format_stamp: "Formatted at 2026-06-19 16:43:20 on dist-test-slave-wvz7"
server_key: "a6ebbfbdfedf87fcc9efa18cd3d9a6df"
server_key_iv: "674a9555fb675045471e3f051ed7c782"
server_key_version: "encryptionkey@0"
I20260619 16:43:20.041885 2216 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887385903558-388-0/minicluster-data/ts-0/wal/instance:
uuid: "0f3f5a7d5afd41ceb534f34d36810852"
format_stamp: "Formatted at 2026-06-19 16:43:20 on dist-test-slave-wvz7"
server_key: "a6ebbfbdfedf87fcc9efa18cd3d9a6df"
server_key_iv: "674a9555fb675045471e3f051ed7c782"
server_key_version: "encryptionkey@0"
I20260619 16:43:20.047456 2216 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260619 16:43:20.051170 2231 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:43:20.052664 2216 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 16:43:20.052807 2216 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887385903558-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887385903558-388-0/minicluster-data/ts-0/wal
uuid: "0f3f5a7d5afd41ceb534f34d36810852"
format_stamp: "Formatted at 2026-06-19 16:43:20 on dist-test-slave-wvz7"
server_key: "a6ebbfbdfedf87fcc9efa18cd3d9a6df"
server_key_iv: "674a9555fb675045471e3f051ed7c782"
server_key_version: "encryptionkey@0"
I20260619 16:43:20.052980 2216 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887385903558-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887385903558-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887385903558-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:43:20.068867 2216 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:43:20.072865 2216 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:43:20.073093 2216 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:43:20.075752 2216 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:43:20.075866 2216 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:43:20.075982 2216 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:43:20.076072 2216 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 16:43:20 dist-test-slave-wvz7 krb5kdc[2100](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887400, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 16:43:20.098384 2193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:20.076269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:57701 (local address 127.0.97.62:46341)
0619 16:43:20.076466 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0619 16:43:20.076473 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:43:20.078060 (+ 1587us) server_negotiation.cc:407] Connection header received
0619 16:43:20.080301 (+ 2241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:43:20.080308 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:43:20.080398 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:43:20.080492 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:43:20.082774 (+ 2282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:20.083760 (+ 986us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:20.086570 (+ 2810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:20.086990 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:20.087095 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:43:20.091828 (+ 4733us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:43:20.091833 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:43:20.091837 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:43:20.091870 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:43:20.095211 (+ 3341us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:43:20.095940 (+ 729us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:43:20.095946 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:43:20.095950 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:43:20.096136 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:43:20.096579 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:43:20.096585 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:43:20.096588 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:43:20.096945 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:43:20.097097 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0619 16:43:20.097775 (+ 678us) server_negotiation.cc:299] Negotiation successful
0619 16:43:20.098005 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260619 16:43:20.099785 2240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:20.076936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46341 (local address 127.0.97.1:57701)
0619 16:43:20.077893 (+ 957us) negotiation.cc:107] Waiting for socket to connect
0619 16:43:20.077946 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0619 16:43:20.080004 (+ 2058us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:43:20.080749 (+ 745us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:43:20.080764 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:43:20.081538 (+ 774us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:43:20.082514 (+ 976us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:43:20.082547 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:43:20.083973 (+ 1426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:43:20.083979 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:43:20.086402 (+ 2423us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:43:20.086421 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:43:20.087232 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:43:20.087240 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:43:20.087542 (+ 302us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:43:20.088304 (+ 762us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:43:20.088337 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:43:20.091589 (+ 3252us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:43:20.095357 (+ 3768us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:43:20.095367 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:43:20.095384 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:43:20.095822 (+ 438us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:43:20.096259 (+ 437us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:43:20.096262 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:43:20.096265 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:43:20.096482 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:43:20.097060 (+ 578us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:43:20.097066 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:43:20.097386 (+ 320us) client_negotiation.cc:769] Sending connection context
0619 16:43:20.097641 (+ 255us) client_negotiation.cc:240] Negotiation successful
0619 16:43:20.098204 (+ 563us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":672,"thread_start_us":423,"threads_started":1}
I20260619 16:43:20.119067 2322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:20.106825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46341 (local address 127.0.97.1:40117)
0619 16:43:20.107382 (+ 557us) negotiation.cc:107] Waiting for socket to connect
0619 16:43:20.107398 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 16:43:20.107539 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:43:20.108002 (+ 463us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:43:20.108006 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:43:20.108480 (+ 474us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:43:20.108754 (+ 274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:43:20.108763 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:43:20.109900 (+ 1137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:43:20.109906 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:43:20.110883 (+ 977us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:43:20.110895 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:43:20.111568 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:43:20.111574 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:43:20.111737 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:43:20.112296 (+ 559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:43:20.112312 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:43:20.113160 (+ 848us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:43:20.116294 (+ 3134us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:43:20.116299 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:43:20.116304 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:43:20.116837 (+ 533us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:43:20.117259 (+ 422us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:43:20.117264 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:43:20.117268 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:43:20.117447 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:43:20.118121 (+ 674us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:43:20.118127 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:43:20.118295 (+ 168us) client_negotiation.cc:769] Sending connection context
0619 16:43:20.118407 (+ 112us) client_negotiation.cc:240] Negotiation successful
0619 16:43:20.118686 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":425,"thread_start_us":290,"threads_started":1}
I20260619 16:43:20.119151 2193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:20.106773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:40117 (local address 127.0.97.62:46341)
0619 16:43:20.106924 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0619 16:43:20.106929 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:43:20.107488 (+ 559us) server_negotiation.cc:407] Connection header received
0619 16:43:20.107653 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:43:20.107659 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:43:20.107726 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:43:20.107832 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:43:20.108917 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:20.109763 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:20.111014 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:20.111399 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:20.111505 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:43:20.113394 (+ 1889us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:43:20.113400 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:43:20.113404 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:43:20.113438 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:43:20.116161 (+ 2723us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:43:20.117000 (+ 839us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:43:20.117006 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:43:20.117009 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:43:20.117163 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:43:20.117566 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:43:20.117572 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:43:20.117575 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:43:20.117970 (+ 395us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:43:20.118171 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0619 16:43:20.118513 (+ 342us) server_negotiation.cc:299] Negotiation successful
0619 16:43:20.118766 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260619 16:43:20.125332 2216 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:35421
I20260619 16:43:20.125357 2350 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:35421 every 8 connection(s)
I20260619 16:43:20.126482 2216 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887385903558-388-0/minicluster-data/ts-0/data/info.pb
I20260619 16:43:20.128381 2351 heartbeater.cc:347] Connected to a master server at 127.0.97.62:46341
I20260619 16:43:20.128635 2351 heartbeater.cc:464] Registering TS with master...
I20260619 16:43:20.129204 2351 heartbeater.cc:511] Master 127.0.97.62:46341 requested a full tablet report, sending...
I20260619 16:43:20.129634 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu as pid 2216
I20260619 16:43:20.129810 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887385903558-388-0/minicluster-data/ts-0/wal/instance
I20260619 16:43:20.130425 388 external_mini_cluster.cc:1428] Setting key 8cc19597d4f5add6e3c58ba6f9f38cf5
I20260619 16:43:20.131309 2145 ts_manager.cc:194] Registered new tserver with Master: 0f3f5a7d5afd41ceb534f34d36810852 (127.0.97.1:35421)
I20260619 16:43:20.133064 2145 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:57701
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887385903558-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887385903558-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260619 16:43:20.166952 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu
/tmp/dist-test-taskEs2EH8/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887385903558-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887385903558-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887385903558-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887385903558-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46341
--keytab_file=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887385903558-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41637
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887385903558-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887385903558-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887385903558-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:43:20.474316 2355 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:43:20.474586 2355 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:43:20.474709 2355 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:43:20.481854 2355 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:43:20.481974 2355 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:43:20.482097 2355 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260619 16:43:20.482156 2355 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 16:43:20.482254 2355 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260619 16:43:20.492005 2355 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41637
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887385903558-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887385903558-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887385903558-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887385903558-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:46341
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887385903558-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:43:20.493295 2355 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:43:20.494601 2355 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:43:20.507486 2360 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:43:20.507462 2363 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:43:20.507747 2361 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:43:20.508287 2355 server_base.cc:1034] running on GCE node
I20260619 16:43:20.508879 2355 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:43:20.509846 2355 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:43:20.511081 2355 hybrid_clock.cc:648] HybridClock initialized: now 1781887400511025 us; error 46 us; skew 500 ppm
I20260619 16:43:20.511408 2355 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:43:20 dist-test-slave-wvz7 krb5kdc[2100](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887400, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:43:20.514389 2355 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260619 16:43:20.515905 2355 webserver.cc:460] Webserver started at http://127.0.97.2:45889/ using document root <none> and password file <none>
I20260619 16:43:20.516536 2355 fs_manager.cc:359] Metadata directory not provided
I20260619 16:43:20.516633 2355 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:43:20.516887 2355 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:43:20.519932 2355 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887385903558-388-0/minicluster-data/ts-1/data/instance:
uuid: "4c9be31194f04e75874e50b6af75e143"
format_stamp: "Formatted at 2026-06-19 16:43:20 on dist-test-slave-wvz7"
server_key: "98b869aac6b683450bddf6c797f7a17e"
server_key_iv: "070248a09ce191667c5571b96fb323e2"
server_key_version: "encryptionkey@0"
I20260619 16:43:20.520641 2355 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887385903558-388-0/minicluster-data/ts-1/wal/instance:
uuid: "4c9be31194f04e75874e50b6af75e143"
format_stamp: "Formatted at 2026-06-19 16:43:20 on dist-test-slave-wvz7"
server_key: "98b869aac6b683450bddf6c797f7a17e"
server_key_iv: "070248a09ce191667c5571b96fb323e2"
server_key_version: "encryptionkey@0"
I20260619 16:43:20.525707 2355 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260619 16:43:20.529096 2370 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:43:20.530572 2355 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 16:43:20.530699 2355 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887385903558-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887385903558-388-0/minicluster-data/ts-1/wal
uuid: "4c9be31194f04e75874e50b6af75e143"
format_stamp: "Formatted at 2026-06-19 16:43:20 on dist-test-slave-wvz7"
server_key: "98b869aac6b683450bddf6c797f7a17e"
server_key_iv: "070248a09ce191667c5571b96fb323e2"
server_key_version: "encryptionkey@0"
I20260619 16:43:20.530859 2355 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887385903558-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887385903558-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887385903558-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:43:20.569315 2355 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:43:20.575129 2355 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:43:20.575491 2355 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:43:20.578786 2355 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:43:20.579010 2355 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260619 16:43:20.579185 2355 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:43:20.579349 2355 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 16:43:20 dist-test-slave-wvz7 krb5kdc[2100](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887400, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 16:43:20.601255 2193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:20.579097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:43399 (local address 127.0.97.62:46341)
0619 16:43:20.579283 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0619 16:43:20.579289 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:43:20.580691 (+ 1402us) server_negotiation.cc:407] Connection header received
0619 16:43:20.582807 (+ 2116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:43:20.582812 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:43:20.582891 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:43:20.582979 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:43:20.584938 (+ 1959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:20.586073 (+ 1135us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:20.587620 (+ 1547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:20.588311 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:20.588524 (+ 213us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:43:20.594495 (+ 5971us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:43:20.594502 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:43:20.594507 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:43:20.594552 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:43:20.597535 (+ 2983us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:43:20.598514 (+ 979us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:43:20.598520 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:43:20.598524 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:43:20.598679 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:43:20.599189 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:43:20.599195 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:43:20.599199 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:43:20.599479 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:43:20.599607 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0619 16:43:20.600610 (+ 1003us) server_negotiation.cc:299] Negotiation successful
0619 16:43:20.600862 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260619 16:43:20.602494 2379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:20.579649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46341 (local address 127.0.97.2:43399)
0619 16:43:20.580529 (+ 880us) negotiation.cc:107] Waiting for socket to connect
0619 16:43:20.580574 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0619 16:43:20.582536 (+ 1962us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:43:20.583242 (+ 706us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:43:20.583254 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:43:20.583850 (+ 596us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:43:20.584718 (+ 868us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:43:20.584747 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:43:20.586209 (+ 1462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:43:20.586214 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:43:20.587480 (+ 1266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:43:20.587492 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:43:20.588936 (+ 1444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:43:20.588944 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:43:20.589270 (+ 326us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:43:20.590244 (+ 974us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:43:20.590278 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:43:20.594115 (+ 3837us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:43:20.597689 (+ 3574us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:43:20.597699 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:43:20.597716 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:43:20.598379 (+ 663us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:43:20.598817 (+ 438us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:43:20.598821 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:43:20.598825 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:43:20.599063 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:43:20.599601 (+ 538us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:43:20.599614 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:43:20.600148 (+ 534us) client_negotiation.cc:769] Sending connection context
0619 16:43:20.600453 (+ 305us) client_negotiation.cc:240] Negotiation successful
0619 16:43:20.600982 (+ 529us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":599,"thread_start_us":369,"threads_started":1}
I20260619 16:43:20.627925 2441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:20.612811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46341 (local address 127.0.97.2:39511)
0619 16:43:20.614151 (+ 1340us) negotiation.cc:107] Waiting for socket to connect
0619 16:43:20.614180 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0619 16:43:20.614383 (+ 203us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:43:20.614773 (+ 390us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:43:20.614778 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:43:20.615323 (+ 545us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:43:20.615654 (+ 331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:43:20.615663 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:43:20.617024 (+ 1361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:43:20.617029 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:43:20.618274 (+ 1245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:43:20.618287 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:43:20.619219 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:43:20.619229 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:43:20.619388 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:43:20.620118 (+ 730us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:43:20.620143 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:43:20.621411 (+ 1268us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:43:20.624691 (+ 3280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:43:20.624700 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:43:20.624704 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:43:20.625345 (+ 641us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:43:20.625923 (+ 578us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:43:20.625927 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:43:20.625929 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:43:20.626052 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:43:20.626695 (+ 643us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:43:20.626700 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:43:20.626877 (+ 177us) client_negotiation.cc:769] Sending connection context
0619 16:43:20.627015 (+ 138us) client_negotiation.cc:240] Negotiation successful
0619 16:43:20.627405 (+ 390us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1163,"thread_start_us":801,"threads_started":1}
I20260619 16:43:20.629241 2193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:20.612664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:39511 (local address 127.0.97.62:46341)
0619 16:43:20.612852 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0619 16:43:20.612864 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 16:43:20.614319 (+ 1455us) server_negotiation.cc:407] Connection header received
0619 16:43:20.614502 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:43:20.614507 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:43:20.614595 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:43:20.614710 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:43:20.615799 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:20.616888 (+ 1089us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:20.618421 (+ 1533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:20.619025 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:20.619154 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:43:20.621613 (+ 2459us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:43:20.621618 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:43:20.621621 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:43:20.621653 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:43:20.624527 (+ 2874us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:43:20.625562 (+ 1035us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:43:20.625580 (+ 18us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:43:20.625584 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:43:20.625755 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:43:20.626170 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:43:20.626176 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:43:20.626179 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:43:20.626583 (+ 404us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:43:20.626715 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0619 16:43:20.628708 (+ 1993us) server_negotiation.cc:299] Negotiation successful
0619 16:43:20.628966 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260619 16:43:20.633342 2355 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:41989
I20260619 16:43:20.633426 2489 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:41989 every 8 connection(s)
I20260619 16:43:20.634656 2355 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887385903558-388-0/minicluster-data/ts-1/data/info.pb
I20260619 16:43:20.636492 2490 heartbeater.cc:347] Connected to a master server at 127.0.97.62:46341
I20260619 16:43:20.636689 2490 heartbeater.cc:464] Registering TS with master...
I20260619 16:43:20.637272 2490 heartbeater.cc:511] Master 127.0.97.62:46341 requested a full tablet report, sending...
I20260619 16:43:20.638504 2145 ts_manager.cc:194] Registered new tserver with Master: 4c9be31194f04e75874e50b6af75e143 (127.0.97.2:41989)
I20260619 16:43:20.639281 2145 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:43399
I20260619 16:43:20.644243 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu as pid 2355
I20260619 16:43:20.644385 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887385903558-388-0/minicluster-data/ts-1/wal/instance
I20260619 16:43:20.644830 388 external_mini_cluster.cc:1428] Setting key b2924380ec9ca96f21f7dcedbddd8b54
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887385903558-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887385903558-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260619 16:43:20.679937 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu
/tmp/dist-test-taskEs2EH8/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887385903558-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887385903558-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887385903558-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887385903558-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46341
--keytab_file=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887385903558-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41637
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887385903558-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887385903558-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887385903558-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:43:20.979146 2494 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:43:20.979425 2494 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:43:20.979564 2494 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:43:20.986582 2494 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:43:20.986696 2494 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:43:20.986820 2494 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260619 16:43:20.986886 2494 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 16:43:20.986994 2494 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260619 16:43:20.996611 2494 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41637
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887385903558-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887385903558-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887385903558-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887385903558-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:46341
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887385903558-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:43:20.997864 2494 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:43:20.999140 2494 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:43:21.011667 2500 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:43:21.011740 2499 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:43:21.011713 2502 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:43:21.015190 2494 server_base.cc:1034] running on GCE node
I20260619 16:43:21.015803 2494 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:43:21.016677 2494 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:43:21.017827 2494 hybrid_clock.cc:648] HybridClock initialized: now 1781887401017776 us; error 45 us; skew 500 ppm
I20260619 16:43:21.018112 2494 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:43:21 dist-test-slave-wvz7 krb5kdc[2100](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887401, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:43:21.020974 2494 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260619 16:43:21.022449 2494 webserver.cc:460] Webserver started at http://127.0.97.3:43337/ using document root <none> and password file <none>
I20260619 16:43:21.023110 2494 fs_manager.cc:359] Metadata directory not provided
I20260619 16:43:21.023221 2494 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:43:21.023488 2494 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:43:21.026268 2494 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887385903558-388-0/minicluster-data/ts-2/data/instance:
uuid: "8cb0bbb22d814658852b0237922bbfb3"
format_stamp: "Formatted at 2026-06-19 16:43:21 on dist-test-slave-wvz7"
server_key: "7189a3f07dcef22d6cbd5da74dbe9730"
server_key_iv: "8e2c8c7e30f87ef91cc863bd84c4d431"
server_key_version: "encryptionkey@0"
I20260619 16:43:21.026969 2494 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887385903558-388-0/minicluster-data/ts-2/wal/instance:
uuid: "8cb0bbb22d814658852b0237922bbfb3"
format_stamp: "Formatted at 2026-06-19 16:43:21 on dist-test-slave-wvz7"
server_key: "7189a3f07dcef22d6cbd5da74dbe9730"
server_key_iv: "8e2c8c7e30f87ef91cc863bd84c4d431"
server_key_version: "encryptionkey@0"
I20260619 16:43:21.031795 2494 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 16:43:21.035087 2509 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:43:21.036489 2494 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 16:43:21.036623 2494 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887385903558-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887385903558-388-0/minicluster-data/ts-2/wal
uuid: "8cb0bbb22d814658852b0237922bbfb3"
format_stamp: "Formatted at 2026-06-19 16:43:21 on dist-test-slave-wvz7"
server_key: "7189a3f07dcef22d6cbd5da74dbe9730"
server_key_iv: "8e2c8c7e30f87ef91cc863bd84c4d431"
server_key_version: "encryptionkey@0"
I20260619 16:43:21.036794 2494 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887385903558-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887385903558-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887385903558-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:43:21.064599 2494 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:43:21.068703 2494 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:43:21.068972 2494 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:43:21.071714 2494 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:43:21.071830 2494 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:43:21.071943 2494 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:43:21.072049 2494 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 16:43:21 dist-test-slave-wvz7 krb5kdc[2100](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887401, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 16:43:21.096195 2518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:21.072906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46341 (local address 127.0.97.3:50009)
0619 16:43:21.073877 (+ 971us) negotiation.cc:107] Waiting for socket to connect
0619 16:43:21.073925 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0619 16:43:21.076064 (+ 2139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:43:21.077184 (+ 1120us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:43:21.077201 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:43:21.078145 (+ 944us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:43:21.079044 (+ 899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:43:21.079072 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:43:21.080413 (+ 1341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:43:21.080422 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:43:21.082218 (+ 1796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:43:21.082233 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:43:21.083152 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:43:21.083159 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:43:21.083630 (+ 471us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:43:21.084466 (+ 836us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:43:21.084511 (+ 45us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:43:21.087809 (+ 3298us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:43:21.091190 (+ 3381us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:43:21.091203 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:43:21.091222 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:43:21.091843 (+ 621us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:43:21.092226 (+ 383us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:43:21.092230 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:43:21.092237 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:43:21.092579 (+ 342us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:43:21.093289 (+ 710us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:43:21.093298 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:43:21.093886 (+ 588us) client_negotiation.cc:769] Sending connection context
0619 16:43:21.094279 (+ 393us) client_negotiation.cc:240] Negotiation successful
0619 16:43:21.094912 (+ 633us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":658,"thread_start_us":420,"threads_started":1}
I20260619 16:43:21.097142 2193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:21.072189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:50009 (local address 127.0.97.62:46341)
0619 16:43:21.072354 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0619 16:43:21.072361 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:43:21.074046 (+ 1685us) server_negotiation.cc:407] Connection header received
0619 16:43:21.076415 (+ 2369us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:43:21.076423 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:43:21.076519 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:43:21.076598 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:43:21.079291 (+ 2693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:21.080269 (+ 978us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:21.082360 (+ 2091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:21.082890 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:21.083038 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:43:21.088080 (+ 5042us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:43:21.088085 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:43:21.088088 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:43:21.088126 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:43:21.091050 (+ 2924us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:43:21.091977 (+ 927us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:43:21.091983 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:43:21.091987 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:43:21.092116 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:43:21.092813 (+ 697us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:43:21.092820 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:43:21.092824 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:43:21.093145 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:43:21.093304 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0619 16:43:21.096710 (+ 3406us) server_negotiation.cc:299] Negotiation successful
0619 16:43:21.096895 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260619 16:43:21.115437 2193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:21.101795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:40005 (local address 127.0.97.62:46341)
0619 16:43:21.101973 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0619 16:43:21.101979 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:43:21.102679 (+ 700us) server_negotiation.cc:407] Connection header received
0619 16:43:21.102941 (+ 262us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:43:21.102946 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:43:21.103019 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:43:21.103097 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:43:21.104516 (+ 1419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:21.105445 (+ 929us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:21.106995 (+ 1550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:21.107576 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:21.107694 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:43:21.109789 (+ 2095us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:43:21.109796 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:43:21.109801 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:43:21.109834 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:43:21.112791 (+ 2957us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:43:21.113540 (+ 749us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:43:21.113545 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:43:21.113549 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:43:21.113757 (+ 208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:43:21.114137 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:43:21.114142 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:43:21.114146 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:43:21.114389 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:43:21.114521 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0619 16:43:21.114875 (+ 354us) server_negotiation.cc:299] Negotiation successful
0619 16:43:21.115057 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260619 16:43:21.115569 2593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:21.101922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46341 (local address 127.0.97.3:40005)
0619 16:43:21.102555 (+ 633us) negotiation.cc:107] Waiting for socket to connect
0619 16:43:21.102582 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0619 16:43:21.102787 (+ 205us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:43:21.103306 (+ 519us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:43:21.103311 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:43:21.103943 (+ 632us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:43:21.104394 (+ 451us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:43:21.104404 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:43:21.105571 (+ 1167us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:43:21.105576 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:43:21.106819 (+ 1243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:43:21.106838 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:43:21.107753 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:43:21.107760 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:43:21.107967 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:43:21.108573 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:43:21.108597 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:43:21.109599 (+ 1002us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:43:21.112911 (+ 3312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:43:21.112916 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:43:21.112920 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:43:21.113425 (+ 505us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:43:21.113883 (+ 458us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:43:21.113889 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:43:21.113893 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:43:21.114038 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:43:21.114491 (+ 453us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:43:21.114495 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:43:21.114655 (+ 160us) client_negotiation.cc:769] Sending connection context
0619 16:43:21.114773 (+ 118us) client_negotiation.cc:240] Negotiation successful
0619 16:43:21.115127 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":495,"thread_start_us":360,"threads_started":1}
I20260619 16:43:21.121940 2494 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:45671
I20260619 16:43:21.121958 2628 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:45671 every 8 connection(s)
I20260619 16:43:21.123121 2494 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887385903558-388-0/minicluster-data/ts-2/data/info.pb
I20260619 16:43:21.125056 2629 heartbeater.cc:347] Connected to a master server at 127.0.97.62:46341
I20260619 16:43:21.125252 2629 heartbeater.cc:464] Registering TS with master...
I20260619 16:43:21.125936 2629 heartbeater.cc:511] Master 127.0.97.62:46341 requested a full tablet report, sending...
I20260619 16:43:21.126031 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu as pid 2494
I20260619 16:43:21.126197 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887385903558-388-0/minicluster-data/ts-2/wal/instance
I20260619 16:43:21.126708 388 external_mini_cluster.cc:1428] Setting key 5ba389da57e4d8074697778d6794bd1a
I20260619 16:43:21.127965 2144 ts_manager.cc:194] Registered new tserver with Master: 8cb0bbb22d814658852b0237922bbfb3 (127.0.97.3:45671)
I20260619 16:43:21.129053 2144 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:50009
I20260619 16:43:21.129937 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 16:43:21.135855 2351 heartbeater.cc:503] Master 127.0.97.62:46341 was elected leader, sending a full tablet report...
Jun 19 16:43:21 dist-test-slave-wvz7 krb5kdc[2100](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887401, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 19 16:43:21 dist-test-slave-wvz7 krb5kdc[2100](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887401, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 16:43:21.172053 2193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:21.159078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59514 (local address 127.0.97.62:46341)
0619 16:43:21.159281 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0619 16:43:21.159286 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:43:21.159425 (+ 139us) server_negotiation.cc:407] Connection header received
0619 16:43:21.159646 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:43:21.159653 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:43:21.159726 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:43:21.159810 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:43:21.161091 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:21.161836 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:21.163036 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:21.163460 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:21.163570 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:43:21.167231 (+ 3661us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:43:21.167236 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:43:21.167239 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:43:21.167276 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:43:21.169609 (+ 2333us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:43:21.170278 (+ 669us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:43:21.170283 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:43:21.170285 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:43:21.170405 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:43:21.170755 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:43:21.170761 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:43:21.170764 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:43:21.171032 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:43:21.171253 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0619 16:43:21.171553 (+ 300us) server_negotiation.cc:299] Negotiation successful
0619 16:43:21.171765 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260619 16:43:21.176014 2144 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59514:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 16:43:21.182662 2144 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 16:43:21.217289 2642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:21.193966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:45671 (local address 127.0.0.1:32966)
0619 16:43:21.201362 (+ 7396us) negotiation.cc:107] Waiting for socket to connect
0619 16:43:21.201419 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0619 16:43:21.201617 (+ 198us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:43:21.209552 (+ 7935us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:43:21.209559 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:43:21.209588 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:43:21.210072 (+ 484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:43:21.210085 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:43:21.212334 (+ 2249us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:43:21.212338 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:43:21.214550 (+ 2212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:43:21.214563 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:43:21.216251 (+ 1688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:43:21.216257 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:43:21.216429 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:43:21.216479 (+ 50us) client_negotiation.cc:769] Sending connection context
0619 16:43:21.216603 (+ 124us) client_negotiation.cc:240] Negotiation successful
0619 16:43:21.216822 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7117,"mutex_wait_us":69,"thread_start_us":6973,"threads_started":1}
I20260619 16:43:21.217381 2644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:21.194811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:41989 (local address 127.0.0.1:33140)
0619 16:43:21.208691 (+ 13880us) negotiation.cc:107] Waiting for socket to connect
0619 16:43:21.208715 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0619 16:43:21.208866 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:43:21.209772 (+ 906us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:43:21.209776 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:43:21.209795 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:43:21.210254 (+ 459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:43:21.210265 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:43:21.212220 (+ 1955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:43:21.212225 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:43:21.215022 (+ 2797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:43:21.215031 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:43:21.216524 (+ 1493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:43:21.216535 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:43:21.216723 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:43:21.216758 (+ 35us) client_negotiation.cc:769] Sending connection context
0619 16:43:21.216843 (+ 85us) client_negotiation.cc:240] Negotiation successful
0619 16:43:21.216979 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13289,"mutex_wait_us":3,"thread_start_us":13139,"threads_started":1}
I20260619 16:43:21.218206 2645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:21.194745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33140 (local address 127.0.97.2:41989)
0619 16:43:21.208534 (+ 13789us) server_negotiation.cc:206] Beginning negotiation
0619 16:43:21.208542 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 16:43:21.208830 (+ 288us) server_negotiation.cc:407] Connection header received
0619 16:43:21.209008 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:43:21.209016 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:43:21.209306 (+ 290us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:43:21.209517 (+ 211us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:43:21.210427 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:21.212039 (+ 1612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:21.215166 (+ 3127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:21.216016 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:21.216147 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:43:21.216359 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0619 16:43:21.217844 (+ 1485us) server_negotiation.cc:299] Negotiation successful
0619 16:43:21.218003 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13638,"thread_start_us":13527,"threads_started":1}
I20260619 16:43:21.218313 2643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:21.194268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32966 (local address 127.0.97.3:45671)
0619 16:43:21.208933 (+ 14665us) server_negotiation.cc:206] Beginning negotiation
0619 16:43:21.208942 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 16:43:21.208966 (+ 24us) server_negotiation.cc:407] Connection header received
0619 16:43:21.209033 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:43:21.209038 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:43:21.209354 (+ 316us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:43:21.210304 (+ 950us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:43:21.210588 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:21.212201 (+ 1613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:21.214685 (+ 2484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:21.216054 (+ 1369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:21.217515 (+ 1461us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:43:21.217780 (+ 265us) server_negotiation.cc:1035] Waiting for connection context
0619 16:43:21.217863 (+ 83us) server_negotiation.cc:299] Negotiation successful
0619 16:43:21.218016 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14508,"thread_start_us":15787,"threads_started":1}
I20260619 16:43:21.220850 2646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:21.202936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:35421 (local address 127.0.0.1:60222)
0619 16:43:21.213578 (+ 10642us) negotiation.cc:107] Waiting for socket to connect
0619 16:43:21.213597 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 16:43:21.213728 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:43:21.214396 (+ 668us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:43:21.214400 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:43:21.214433 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:43:21.214835 (+ 402us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:43:21.214844 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:43:21.216585 (+ 1741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:43:21.216590 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:43:21.218600 (+ 2010us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:43:21.218613 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:43:21.220146 (+ 1533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:43:21.220152 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:43:21.220309 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:43:21.220338 (+ 29us) client_negotiation.cc:769] Sending connection context
0619 16:43:21.220423 (+ 85us) client_negotiation.cc:240] Negotiation successful
0619 16:43:21.220547 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10515,"thread_start_us":10359,"threads_started":1}
I20260619 16:43:21.220896 2647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:21.203136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60222 (local address 127.0.97.1:35421)
0619 16:43:21.213194 (+ 10058us) server_negotiation.cc:206] Beginning negotiation
0619 16:43:21.213204 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 16:43:21.213708 (+ 504us) server_negotiation.cc:407] Connection header received
0619 16:43:21.213925 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:43:21.213933 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:43:21.214200 (+ 267us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:43:21.214385 (+ 185us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:43:21.215024 (+ 639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:21.216431 (+ 1407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:21.218720 (+ 2289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:21.219958 (+ 1238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:21.220088 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:43:21.220263 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0619 16:43:21.220493 (+ 230us) server_negotiation.cc:299] Negotiation successful
0619 16:43:21.220642 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9904,"thread_start_us":9755,"threads_started":1}
I20260619 16:43:21.221675 2420 tablet_service.cc:1467] Processing CreateTablet for tablet b0bce0c710c54d96af5f8113c8ab6a33 (DEFAULT_TABLE table=test-table [id=2aa280e7c7e44871a863c85bbcbb00b6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:43:21.221726 2559 tablet_service.cc:1467] Processing CreateTablet for tablet b0bce0c710c54d96af5f8113c8ab6a33 (DEFAULT_TABLE table=test-table [id=2aa280e7c7e44871a863c85bbcbb00b6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:43:21.223058 2559 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b0bce0c710c54d96af5f8113c8ab6a33. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:43:21.223088 2420 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b0bce0c710c54d96af5f8113c8ab6a33. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:43:21.224882 2281 tablet_service.cc:1467] Processing CreateTablet for tablet b0bce0c710c54d96af5f8113c8ab6a33 (DEFAULT_TABLE table=test-table [id=2aa280e7c7e44871a863c85bbcbb00b6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:43:21.226118 2281 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b0bce0c710c54d96af5f8113c8ab6a33. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:43:21.233878 2648 tablet_bootstrap.cc:492] T b0bce0c710c54d96af5f8113c8ab6a33 P 8cb0bbb22d814658852b0237922bbfb3: Bootstrap starting.
I20260619 16:43:21.236626 2649 tablet_bootstrap.cc:492] T b0bce0c710c54d96af5f8113c8ab6a33 P 4c9be31194f04e75874e50b6af75e143: Bootstrap starting.
I20260619 16:43:21.237344 2648 tablet_bootstrap.cc:654] T b0bce0c710c54d96af5f8113c8ab6a33 P 8cb0bbb22d814658852b0237922bbfb3: Neither blocks nor log segments found. Creating new log.
I20260619 16:43:21.238632 2648 log.cc:824] T b0bce0c710c54d96af5f8113c8ab6a33 P 8cb0bbb22d814658852b0237922bbfb3: Log is configured to *not* fsync() on all Append() calls
I20260619 16:43:21.239140 2650 tablet_bootstrap.cc:492] T b0bce0c710c54d96af5f8113c8ab6a33 P 0f3f5a7d5afd41ceb534f34d36810852: Bootstrap starting.
I20260619 16:43:21.239943 2649 tablet_bootstrap.cc:654] T b0bce0c710c54d96af5f8113c8ab6a33 P 4c9be31194f04e75874e50b6af75e143: Neither blocks nor log segments found. Creating new log.
I20260619 16:43:21.241283 2649 log.cc:824] T b0bce0c710c54d96af5f8113c8ab6a33 P 4c9be31194f04e75874e50b6af75e143: Log is configured to *not* fsync() on all Append() calls
I20260619 16:43:21.241991 2650 tablet_bootstrap.cc:654] T b0bce0c710c54d96af5f8113c8ab6a33 P 0f3f5a7d5afd41ceb534f34d36810852: Neither blocks nor log segments found. Creating new log.
I20260619 16:43:21.243184 2650 log.cc:824] T b0bce0c710c54d96af5f8113c8ab6a33 P 0f3f5a7d5afd41ceb534f34d36810852: Log is configured to *not* fsync() on all Append() calls
I20260619 16:43:21.244199 2648 tablet_bootstrap.cc:492] T b0bce0c710c54d96af5f8113c8ab6a33 P 8cb0bbb22d814658852b0237922bbfb3: No bootstrap required, opened a new log
I20260619 16:43:21.244247 2649 tablet_bootstrap.cc:492] T b0bce0c710c54d96af5f8113c8ab6a33 P 4c9be31194f04e75874e50b6af75e143: No bootstrap required, opened a new log
I20260619 16:43:21.244432 2648 ts_tablet_manager.cc:1397] T b0bce0c710c54d96af5f8113c8ab6a33 P 8cb0bbb22d814658852b0237922bbfb3: Time spent bootstrapping tablet: real 0.011s user 0.008s sys 0.000s
I20260619 16:43:21.244498 2649 ts_tablet_manager.cc:1397] T b0bce0c710c54d96af5f8113c8ab6a33 P 4c9be31194f04e75874e50b6af75e143: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260619 16:43:21.246160 2650 tablet_bootstrap.cc:492] T b0bce0c710c54d96af5f8113c8ab6a33 P 0f3f5a7d5afd41ceb534f34d36810852: No bootstrap required, opened a new log
I20260619 16:43:21.246390 2650 ts_tablet_manager.cc:1397] T b0bce0c710c54d96af5f8113c8ab6a33 P 0f3f5a7d5afd41ceb534f34d36810852: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260619 16:43:21.251133 2649 raft_consensus.cc:348] T b0bce0c710c54d96af5f8113c8ab6a33 P 4c9be31194f04e75874e50b6af75e143 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8cb0bbb22d814658852b0237922bbfb3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45671 } } peers { permanent_uuid: "0f3f5a7d5afd41ceb534f34d36810852" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35421 } } peers { permanent_uuid: "4c9be31194f04e75874e50b6af75e143" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41989 } }
I20260619 16:43:21.251488 2649 raft_consensus.cc:374] T b0bce0c710c54d96af5f8113c8ab6a33 P 4c9be31194f04e75874e50b6af75e143 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:43:21.251596 2649 raft_consensus.cc:729] T b0bce0c710c54d96af5f8113c8ab6a33 P 4c9be31194f04e75874e50b6af75e143 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c9be31194f04e75874e50b6af75e143, State: Initialized, Role: FOLLOWER
I20260619 16:43:21.252143 2649 consensus_queue.cc:260] T b0bce0c710c54d96af5f8113c8ab6a33 P 4c9be31194f04e75874e50b6af75e143 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8cb0bbb22d814658852b0237922bbfb3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45671 } } peers { permanent_uuid: "0f3f5a7d5afd41ceb534f34d36810852" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35421 } } peers { permanent_uuid: "4c9be31194f04e75874e50b6af75e143" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41989 } }
I20260619 16:43:21.253093 2490 heartbeater.cc:503] Master 127.0.97.62:46341 was elected leader, sending a full tablet report...
I20260619 16:43:21.254421 2648 raft_consensus.cc:348] T b0bce0c710c54d96af5f8113c8ab6a33 P 8cb0bbb22d814658852b0237922bbfb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8cb0bbb22d814658852b0237922bbfb3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45671 } } peers { permanent_uuid: "0f3f5a7d5afd41ceb534f34d36810852" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35421 } } peers { permanent_uuid: "4c9be31194f04e75874e50b6af75e143" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41989 } }
I20260619 16:43:21.254873 2648 raft_consensus.cc:374] T b0bce0c710c54d96af5f8113c8ab6a33 P 8cb0bbb22d814658852b0237922bbfb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:43:21.254971 2649 ts_tablet_manager.cc:1428] T b0bce0c710c54d96af5f8113c8ab6a33 P 4c9be31194f04e75874e50b6af75e143: Time spent starting tablet: real 0.010s user 0.008s sys 0.000s
I20260619 16:43:21.254977 2648 raft_consensus.cc:729] T b0bce0c710c54d96af5f8113c8ab6a33 P 8cb0bbb22d814658852b0237922bbfb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8cb0bbb22d814658852b0237922bbfb3, State: Initialized, Role: FOLLOWER
I20260619 16:43:21.256024 2648 consensus_queue.cc:260] T b0bce0c710c54d96af5f8113c8ab6a33 P 8cb0bbb22d814658852b0237922bbfb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8cb0bbb22d814658852b0237922bbfb3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45671 } } peers { permanent_uuid: "0f3f5a7d5afd41ceb534f34d36810852" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35421 } } peers { permanent_uuid: "4c9be31194f04e75874e50b6af75e143" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41989 } }
I20260619 16:43:21.256580 2650 raft_consensus.cc:348] T b0bce0c710c54d96af5f8113c8ab6a33 P 0f3f5a7d5afd41ceb534f34d36810852 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8cb0bbb22d814658852b0237922bbfb3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45671 } } peers { permanent_uuid: "0f3f5a7d5afd41ceb534f34d36810852" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35421 } } peers { permanent_uuid: "4c9be31194f04e75874e50b6af75e143" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41989 } }
I20260619 16:43:21.257030 2650 raft_consensus.cc:374] T b0bce0c710c54d96af5f8113c8ab6a33 P 0f3f5a7d5afd41ceb534f34d36810852 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:43:21.257309 2629 heartbeater.cc:503] Master 127.0.97.62:46341 was elected leader, sending a full tablet report...
I20260619 16:43:21.258003 2648 ts_tablet_manager.cc:1428] T b0bce0c710c54d96af5f8113c8ab6a33 P 8cb0bbb22d814658852b0237922bbfb3: Time spent starting tablet: real 0.013s user 0.015s sys 0.000s
I20260619 16:43:21.258028 2650 raft_consensus.cc:729] T b0bce0c710c54d96af5f8113c8ab6a33 P 0f3f5a7d5afd41ceb534f34d36810852 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f3f5a7d5afd41ceb534f34d36810852, State: Initialized, Role: FOLLOWER
I20260619 16:43:21.258679 2650 consensus_queue.cc:260] T b0bce0c710c54d96af5f8113c8ab6a33 P 0f3f5a7d5afd41ceb534f34d36810852 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8cb0bbb22d814658852b0237922bbfb3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45671 } } peers { permanent_uuid: "0f3f5a7d5afd41ceb534f34d36810852" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35421 } } peers { permanent_uuid: "4c9be31194f04e75874e50b6af75e143" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41989 } }
I20260619 16:43:21.261010 2650 ts_tablet_manager.cc:1428] T b0bce0c710c54d96af5f8113c8ab6a33 P 0f3f5a7d5afd41ceb534f34d36810852: Time spent starting tablet: real 0.015s user 0.013s sys 0.000s
I20260619 16:43:21.354619 2655 raft_consensus.cc:482] T b0bce0c710c54d96af5f8113c8ab6a33 P 8cb0bbb22d814658852b0237922bbfb3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 16:43:21.354950 2655 raft_consensus.cc:504] T b0bce0c710c54d96af5f8113c8ab6a33 P 8cb0bbb22d814658852b0237922bbfb3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8cb0bbb22d814658852b0237922bbfb3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45671 } } peers { permanent_uuid: "0f3f5a7d5afd41ceb534f34d36810852" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35421 } } peers { permanent_uuid: "4c9be31194f04e75874e50b6af75e143" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41989 } }
I20260619 16:43:21.356698 2655 leader_election.cc:288] T b0bce0c710c54d96af5f8113c8ab6a33 P 8cb0bbb22d814658852b0237922bbfb3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0f3f5a7d5afd41ceb534f34d36810852 (127.0.97.1:35421), 4c9be31194f04e75874e50b6af75e143 (127.0.97.2:41989)
I20260619 16:43:21.361925 2647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:21.357106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:45405 (local address 127.0.97.1:35421)
0619 16:43:21.357299 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0619 16:43:21.357306 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:43:21.357327 (+ 21us) server_negotiation.cc:407] Connection header received
0619 16:43:21.357399 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:43:21.357404 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:43:21.357486 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:43:21.357581 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:43:21.358155 (+ 574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:21.358830 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:21.360354 (+ 1524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:21.361115 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:21.361233 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:43:21.361328 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0619 16:43:21.361563 (+ 235us) server_negotiation.cc:299] Negotiation successful
0619 16:43:21.361684 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260619 16:43:21.361965 2518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:21.356922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:35421 (local address 127.0.97.3:45405)
0619 16:43:21.357117 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0619 16:43:21.357139 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0619 16:43:21.357260 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:43:21.357795 (+ 535us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:43:21.357800 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:43:21.357836 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:43:21.358015 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:43:21.358025 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:43:21.358963 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:43:21.358968 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:43:21.360195 (+ 1227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:43:21.360206 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:43:21.361261 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:43:21.361267 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:43:21.361423 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:43:21.361446 (+ 23us) client_negotiation.cc:769] Sending connection context
0619 16:43:21.361527 (+ 81us) client_negotiation.cc:240] Negotiation successful
0619 16:43:21.361651 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48}
I20260619 16:43:21.362506 2657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:21.357232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:41989 (local address 127.0.97.3:40999)
0619 16:43:21.357961 (+ 729us) negotiation.cc:107] Waiting for socket to connect
0619 16:43:21.357975 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 16:43:21.358085 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:43:21.358341 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:43:21.358345 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:43:21.358365 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:43:21.358643 (+ 278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:43:21.358650 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:43:21.359565 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:43:21.359573 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:43:21.360827 (+ 1254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:43:21.360844 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:43:21.361894 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:43:21.361900 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:43:21.362048 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:43:21.362083 (+ 35us) client_negotiation.cc:769] Sending connection context
0619 16:43:21.362171 (+ 88us) client_negotiation.cc:240] Negotiation successful
0619 16:43:21.362280 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":620,"thread_start_us":398,"threads_started":1}
I20260619 16:43:21.362577 2645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:21.357530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:40999 (local address 127.0.97.2:41989)
0619 16:43:21.357758 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0619 16:43:21.357763 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:43:21.358061 (+ 298us) server_negotiation.cc:407] Connection header received
0619 16:43:21.358201 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:43:21.358204 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:43:21.358254 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:43:21.358321 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:43:21.358786 (+ 465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:21.359448 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:21.360972 (+ 1524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:21.361674 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:21.361811 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:43:21.361903 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0619 16:43:21.362218 (+ 315us) server_negotiation.cc:299] Negotiation successful
0619 16:43:21.362345 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260619 16:43:21.363056 2301 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0bce0c710c54d96af5f8113c8ab6a33" candidate_uuid: "8cb0bbb22d814658852b0237922bbfb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f3f5a7d5afd41ceb534f34d36810852" is_pre_election: true
I20260619 16:43:21.363510 2301 raft_consensus.cc:2454] T b0bce0c710c54d96af5f8113c8ab6a33 P 0f3f5a7d5afd41ceb534f34d36810852 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8cb0bbb22d814658852b0237922bbfb3 in term 0.
I20260619 16:43:21.363523 2445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0bce0c710c54d96af5f8113c8ab6a33" candidate_uuid: "8cb0bbb22d814658852b0237922bbfb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c9be31194f04e75874e50b6af75e143" is_pre_election: true
I20260619 16:43:21.363967 2445 raft_consensus.cc:2454] T b0bce0c710c54d96af5f8113c8ab6a33 P 4c9be31194f04e75874e50b6af75e143 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8cb0bbb22d814658852b0237922bbfb3 in term 0.
I20260619 16:43:21.364144 2510 leader_election.cc:302] T b0bce0c710c54d96af5f8113c8ab6a33 P 8cb0bbb22d814658852b0237922bbfb3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f3f5a7d5afd41ceb534f34d36810852, 8cb0bbb22d814658852b0237922bbfb3; no voters:
I20260619 16:43:21.364491 2655 raft_consensus.cc:2781] T b0bce0c710c54d96af5f8113c8ab6a33 P 8cb0bbb22d814658852b0237922bbfb3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 16:43:21.364612 2655 raft_consensus.cc:482] T b0bce0c710c54d96af5f8113c8ab6a33 P 8cb0bbb22d814658852b0237922bbfb3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 16:43:21.364696 2655 raft_consensus.cc:3037] T b0bce0c710c54d96af5f8113c8ab6a33 P 8cb0bbb22d814658852b0237922bbfb3 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:43:21.366843 2655 raft_consensus.cc:504] T b0bce0c710c54d96af5f8113c8ab6a33 P 8cb0bbb22d814658852b0237922bbfb3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8cb0bbb22d814658852b0237922bbfb3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45671 } } peers { permanent_uuid: "0f3f5a7d5afd41ceb534f34d36810852" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35421 } } peers { permanent_uuid: "4c9be31194f04e75874e50b6af75e143" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41989 } }
I20260619 16:43:21.367381 2655 leader_election.cc:288] T b0bce0c710c54d96af5f8113c8ab6a33 P 8cb0bbb22d814658852b0237922bbfb3 [CANDIDATE]: Term 1 election: Requested vote from peers 0f3f5a7d5afd41ceb534f34d36810852 (127.0.97.1:35421), 4c9be31194f04e75874e50b6af75e143 (127.0.97.2:41989)
I20260619 16:43:21.367784 2301 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0bce0c710c54d96af5f8113c8ab6a33" candidate_uuid: "8cb0bbb22d814658852b0237922bbfb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f3f5a7d5afd41ceb534f34d36810852"
I20260619 16:43:21.367821 2445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0bce0c710c54d96af5f8113c8ab6a33" candidate_uuid: "8cb0bbb22d814658852b0237922bbfb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c9be31194f04e75874e50b6af75e143"
I20260619 16:43:21.367949 2301 raft_consensus.cc:3037] T b0bce0c710c54d96af5f8113c8ab6a33 P 0f3f5a7d5afd41ceb534f34d36810852 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:43:21.367985 2445 raft_consensus.cc:3037] T b0bce0c710c54d96af5f8113c8ab6a33 P 4c9be31194f04e75874e50b6af75e143 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:43:21.370018 2301 raft_consensus.cc:2454] T b0bce0c710c54d96af5f8113c8ab6a33 P 0f3f5a7d5afd41ceb534f34d36810852 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8cb0bbb22d814658852b0237922bbfb3 in term 1.
I20260619 16:43:21.370030 2445 raft_consensus.cc:2454] T b0bce0c710c54d96af5f8113c8ab6a33 P 4c9be31194f04e75874e50b6af75e143 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8cb0bbb22d814658852b0237922bbfb3 in term 1.
I20260619 16:43:21.370375 2511 leader_election.cc:302] T b0bce0c710c54d96af5f8113c8ab6a33 P 8cb0bbb22d814658852b0237922bbfb3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c9be31194f04e75874e50b6af75e143, 8cb0bbb22d814658852b0237922bbfb3; no voters:
I20260619 16:43:21.370662 2655 raft_consensus.cc:2781] T b0bce0c710c54d96af5f8113c8ab6a33 P 8cb0bbb22d814658852b0237922bbfb3 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:43:21.371047 2655 raft_consensus.cc:686] T b0bce0c710c54d96af5f8113c8ab6a33 P 8cb0bbb22d814658852b0237922bbfb3 [term 1 LEADER]: Becoming Leader. State: Replica: 8cb0bbb22d814658852b0237922bbfb3, State: Running, Role: LEADER
I20260619 16:43:21.371474 2655 consensus_queue.cc:237] T b0bce0c710c54d96af5f8113c8ab6a33 P 8cb0bbb22d814658852b0237922bbfb3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8cb0bbb22d814658852b0237922bbfb3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45671 } } peers { permanent_uuid: "0f3f5a7d5afd41ceb534f34d36810852" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35421 } } peers { permanent_uuid: "4c9be31194f04e75874e50b6af75e143" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41989 } }
W20260619 16:43:21.374558 2630 tablet.cc:2369] T b0bce0c710c54d96af5f8113c8ab6a33 P 8cb0bbb22d814658852b0237922bbfb3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 16:43:21.375649 2144 catalog_manager.cc:5495] T b0bce0c710c54d96af5f8113c8ab6a33 P 8cb0bbb22d814658852b0237922bbfb3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8cb0bbb22d814658852b0237922bbfb3 (127.0.97.3). New cstate: current_term: 1 leader_uuid: "8cb0bbb22d814658852b0237922bbfb3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8cb0bbb22d814658852b0237922bbfb3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45671 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0f3f5a7d5afd41ceb534f34d36810852" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35421 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4c9be31194f04e75874e50b6af75e143" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41989 } health_report { overall_health: UNKNOWN } } }
W20260619 16:43:21.379176 2352 tablet.cc:2369] T b0bce0c710c54d96af5f8113c8ab6a33 P 0f3f5a7d5afd41ceb534f34d36810852: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 16:43:21.386428 2491 tablet.cc:2369] T b0bce0c710c54d96af5f8113c8ab6a33 P 4c9be31194f04e75874e50b6af75e143: Can't schedule compaction. Clean time has not been advanced past its initial value.
Jun 19 16:43:21 dist-test-slave-wvz7 krb5kdc[2100](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887399, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 16:43:21.428728 2665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:21.416109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46341 (local address 127.0.0.1:59516)
0619 16:43:21.416639 (+ 530us) negotiation.cc:107] Waiting for socket to connect
0619 16:43:21.416654 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 16:43:21.416788 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:43:21.417070 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:43:21.417074 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:43:21.417778 (+ 704us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:43:21.418016 (+ 238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:43:21.418025 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:43:21.418962 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:43:21.418967 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:43:21.419907 (+ 940us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:43:21.419917 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:43:21.420598 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:43:21.420605 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:43:21.420755 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:43:21.421302 (+ 547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:43:21.421318 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:43:21.423504 (+ 2186us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:43:21.426412 (+ 2908us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:43:21.426425 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:43:21.426433 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:43:21.426836 (+ 403us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:43:21.427160 (+ 324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:43:21.427164 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:43:21.427168 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:43:21.427281 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:43:21.427742 (+ 461us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:43:21.427750 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:43:21.427901 (+ 151us) client_negotiation.cc:769] Sending connection context
0619 16:43:21.428039 (+ 138us) client_negotiation.cc:240] Negotiation successful
0619 16:43:21.428289 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":415,"thread_start_us":285,"threads_started":1}
I20260619 16:43:21.428748 2193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:21.416044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59516 (local address 127.0.97.62:46341)
0619 16:43:21.416215 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0619 16:43:21.416219 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:43:21.416735 (+ 516us) server_negotiation.cc:407] Connection header received
0619 16:43:21.416890 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:43:21.416894 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:43:21.416945 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:43:21.417018 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:43:21.418143 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:21.418847 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:21.420023 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:21.420425 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:21.420525 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:43:21.423705 (+ 3180us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:43:21.423710 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:43:21.423714 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:43:21.423743 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:43:21.426289 (+ 2546us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:43:21.426940 (+ 651us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:43:21.426945 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:43:21.426948 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:43:21.427068 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:43:21.427371 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:43:21.427376 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:43:21.427387 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:43:21.427623 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:43:21.427743 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0619 16:43:21.428127 (+ 384us) server_negotiation.cc:299] Negotiation successful
0619 16:43:21.428430 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":107,"server-negotiator.queue_time_us":52}
I20260619 16:43:21.433461 2144 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:59516:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260619 16:43:21.434461 2144 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 16:43:21.440634 2281 tablet_service.cc:1467] Processing CreateTablet for tablet e65e2817c4d54ce1b545235ebf8040fb (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=50b7a481850a4299ad9b59dbfaa76336]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 16:43:21.440797 2420 tablet_service.cc:1467] Processing CreateTablet for tablet e65e2817c4d54ce1b545235ebf8040fb (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=50b7a481850a4299ad9b59dbfaa76336]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 16:43:21.440801 2559 tablet_service.cc:1467] Processing CreateTablet for tablet e65e2817c4d54ce1b545235ebf8040fb (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=50b7a481850a4299ad9b59dbfaa76336]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 16:43:21.441447 2281 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e65e2817c4d54ce1b545235ebf8040fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:43:21.441641 2559 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e65e2817c4d54ce1b545235ebf8040fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:43:21.441641 2420 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e65e2817c4d54ce1b545235ebf8040fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:43:21.446097 2650 tablet_bootstrap.cc:492] T e65e2817c4d54ce1b545235ebf8040fb P 0f3f5a7d5afd41ceb534f34d36810852: Bootstrap starting.
I20260619 16:43:21.447497 2650 tablet_bootstrap.cc:654] T e65e2817c4d54ce1b545235ebf8040fb P 0f3f5a7d5afd41ceb534f34d36810852: Neither blocks nor log segments found. Creating new log.
I20260619 16:43:21.448257 2648 tablet_bootstrap.cc:492] T e65e2817c4d54ce1b545235ebf8040fb P 8cb0bbb22d814658852b0237922bbfb3: Bootstrap starting.
I20260619 16:43:21.448552 2649 tablet_bootstrap.cc:492] T e65e2817c4d54ce1b545235ebf8040fb P 4c9be31194f04e75874e50b6af75e143: Bootstrap starting.
I20260619 16:43:21.449640 2649 tablet_bootstrap.cc:654] T e65e2817c4d54ce1b545235ebf8040fb P 4c9be31194f04e75874e50b6af75e143: Neither blocks nor log segments found. Creating new log.
I20260619 16:43:21.449858 2648 tablet_bootstrap.cc:654] T e65e2817c4d54ce1b545235ebf8040fb P 8cb0bbb22d814658852b0237922bbfb3: Neither blocks nor log segments found. Creating new log.
I20260619 16:43:21.451545 2649 tablet_bootstrap.cc:492] T e65e2817c4d54ce1b545235ebf8040fb P 4c9be31194f04e75874e50b6af75e143: No bootstrap required, opened a new log
I20260619 16:43:21.451666 2649 ts_tablet_manager.cc:1397] T e65e2817c4d54ce1b545235ebf8040fb P 4c9be31194f04e75874e50b6af75e143: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260619 16:43:21.452226 2649 raft_consensus.cc:348] T e65e2817c4d54ce1b545235ebf8040fb P 4c9be31194f04e75874e50b6af75e143 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f3f5a7d5afd41ceb534f34d36810852" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35421 } } peers { permanent_uuid: "4c9be31194f04e75874e50b6af75e143" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41989 } } peers { permanent_uuid: "8cb0bbb22d814658852b0237922bbfb3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45671 } }
I20260619 16:43:21.452399 2649 raft_consensus.cc:374] T e65e2817c4d54ce1b545235ebf8040fb P 4c9be31194f04e75874e50b6af75e143 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:43:21.452479 2649 raft_consensus.cc:729] T e65e2817c4d54ce1b545235ebf8040fb P 4c9be31194f04e75874e50b6af75e143 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c9be31194f04e75874e50b6af75e143, State: Initialized, Role: FOLLOWER
I20260619 16:43:21.452688 2649 consensus_queue.cc:260] T e65e2817c4d54ce1b545235ebf8040fb P 4c9be31194f04e75874e50b6af75e143 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f3f5a7d5afd41ceb534f34d36810852" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35421 } } peers { permanent_uuid: "4c9be31194f04e75874e50b6af75e143" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41989 } } peers { permanent_uuid: "8cb0bbb22d814658852b0237922bbfb3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45671 } }
I20260619 16:43:21.453188 2649 ts_tablet_manager.cc:1428] T e65e2817c4d54ce1b545235ebf8040fb P 4c9be31194f04e75874e50b6af75e143: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260619 16:43:21.453094 2654 tablet_replica.cc:439] T e65e2817c4d54ce1b545235ebf8040fb P 4c9be31194f04e75874e50b6af75e143: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f3f5a7d5afd41ceb534f34d36810852" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35421 } } peers { permanent_uuid: "4c9be31194f04e75874e50b6af75e143" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41989 } } peers { permanent_uuid: "8cb0bbb22d814658852b0237922bbfb3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45671 } } }
I20260619 16:43:21.453424 2654 tablet_replica.cc:442] T e65e2817c4d54ce1b545235ebf8040fb P 4c9be31194f04e75874e50b6af75e143: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 16:43:21.454773 2648 tablet_bootstrap.cc:492] T e65e2817c4d54ce1b545235ebf8040fb P 8cb0bbb22d814658852b0237922bbfb3: No bootstrap required, opened a new log
I20260619 16:43:21.454912 2648 ts_tablet_manager.cc:1397] T e65e2817c4d54ce1b545235ebf8040fb P 8cb0bbb22d814658852b0237922bbfb3: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260619 16:43:21.455024 2650 tablet_bootstrap.cc:492] T e65e2817c4d54ce1b545235ebf8040fb P 0f3f5a7d5afd41ceb534f34d36810852: No bootstrap required, opened a new log
I20260619 16:43:21.455175 2650 ts_tablet_manager.cc:1397] T e65e2817c4d54ce1b545235ebf8040fb P 0f3f5a7d5afd41ceb534f34d36810852: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.005s
I20260619 16:43:21.455703 2648 raft_consensus.cc:348] T e65e2817c4d54ce1b545235ebf8040fb P 8cb0bbb22d814658852b0237922bbfb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f3f5a7d5afd41ceb534f34d36810852" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35421 } } peers { permanent_uuid: "4c9be31194f04e75874e50b6af75e143" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41989 } } peers { permanent_uuid: "8cb0bbb22d814658852b0237922bbfb3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45671 } }
I20260619 16:43:21.455974 2648 raft_consensus.cc:374] T e65e2817c4d54ce1b545235ebf8040fb P 8cb0bbb22d814658852b0237922bbfb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:43:21.456063 2648 raft_consensus.cc:729] T e65e2817c4d54ce1b545235ebf8040fb P 8cb0bbb22d814658852b0237922bbfb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8cb0bbb22d814658852b0237922bbfb3, State: Initialized, Role: FOLLOWER
I20260619 16:43:21.455935 2650 raft_consensus.cc:348] T e65e2817c4d54ce1b545235ebf8040fb P 0f3f5a7d5afd41ceb534f34d36810852 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f3f5a7d5afd41ceb534f34d36810852" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35421 } } peers { permanent_uuid: "4c9be31194f04e75874e50b6af75e143" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41989 } } peers { permanent_uuid: "8cb0bbb22d814658852b0237922bbfb3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45671 } }
I20260619 16:43:21.456147 2650 raft_consensus.cc:374] T e65e2817c4d54ce1b545235ebf8040fb P 0f3f5a7d5afd41ceb534f34d36810852 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:43:21.456230 2650 raft_consensus.cc:729] T e65e2817c4d54ce1b545235ebf8040fb P 0f3f5a7d5afd41ceb534f34d36810852 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f3f5a7d5afd41ceb534f34d36810852, State: Initialized, Role: FOLLOWER
I20260619 16:43:21.456279 2648 consensus_queue.cc:260] T e65e2817c4d54ce1b545235ebf8040fb P 8cb0bbb22d814658852b0237922bbfb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f3f5a7d5afd41ceb534f34d36810852" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35421 } } peers { permanent_uuid: "4c9be31194f04e75874e50b6af75e143" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41989 } } peers { permanent_uuid: "8cb0bbb22d814658852b0237922bbfb3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45671 } }
I20260619 16:43:21.456424 2650 consensus_queue.cc:260] T e65e2817c4d54ce1b545235ebf8040fb P 0f3f5a7d5afd41ceb534f34d36810852 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f3f5a7d5afd41ceb534f34d36810852" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35421 } } peers { permanent_uuid: "4c9be31194f04e75874e50b6af75e143" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41989 } } peers { permanent_uuid: "8cb0bbb22d814658852b0237922bbfb3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45671 } }
I20260619 16:43:21.456708 2655 tablet_replica.cc:439] T e65e2817c4d54ce1b545235ebf8040fb P 8cb0bbb22d814658852b0237922bbfb3: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f3f5a7d5afd41ceb534f34d36810852" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35421 } } peers { permanent_uuid: "4c9be31194f04e75874e50b6af75e143" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41989 } } peers { permanent_uuid: "8cb0bbb22d814658852b0237922bbfb3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45671 } } }
I20260619 16:43:21.456995 2655 tablet_replica.cc:442] T e65e2817c4d54ce1b545235ebf8040fb P 8cb0bbb22d814658852b0237922bbfb3: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 16:43:21.457012 2650 ts_tablet_manager.cc:1428] T e65e2817c4d54ce1b545235ebf8040fb P 0f3f5a7d5afd41ceb534f34d36810852: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260619 16:43:21.457202 2656 tablet_replica.cc:439] T e65e2817c4d54ce1b545235ebf8040fb P 0f3f5a7d5afd41ceb534f34d36810852: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f3f5a7d5afd41ceb534f34d36810852" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35421 } } peers { permanent_uuid: "4c9be31194f04e75874e50b6af75e143" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41989 } } peers { permanent_uuid: "8cb0bbb22d814658852b0237922bbfb3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45671 } } }
I20260619 16:43:21.457475 2656 tablet_replica.cc:442] T e65e2817c4d54ce1b545235ebf8040fb P 0f3f5a7d5afd41ceb534f34d36810852: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 16:43:21.456853 2648 ts_tablet_manager.cc:1428] T e65e2817c4d54ce1b545235ebf8040fb P 8cb0bbb22d814658852b0237922bbfb3: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260619 16:43:21.629484 2655 raft_consensus.cc:482] T e65e2817c4d54ce1b545235ebf8040fb P 8cb0bbb22d814658852b0237922bbfb3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 16:43:21.629676 2655 raft_consensus.cc:504] T e65e2817c4d54ce1b545235ebf8040fb P 8cb0bbb22d814658852b0237922bbfb3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f3f5a7d5afd41ceb534f34d36810852" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35421 } } peers { permanent_uuid: "4c9be31194f04e75874e50b6af75e143" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41989 } } peers { permanent_uuid: "8cb0bbb22d814658852b0237922bbfb3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45671 } }
I20260619 16:43:21.630362 2655 leader_election.cc:288] T e65e2817c4d54ce1b545235ebf8040fb P 8cb0bbb22d814658852b0237922bbfb3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0f3f5a7d5afd41ceb534f34d36810852 (127.0.97.1:35421), 4c9be31194f04e75874e50b6af75e143 (127.0.97.2:41989)
I20260619 16:43:21.630932 2445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e65e2817c4d54ce1b545235ebf8040fb" candidate_uuid: "8cb0bbb22d814658852b0237922bbfb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c9be31194f04e75874e50b6af75e143" is_pre_election: true
I20260619 16:43:21.630935 2301 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e65e2817c4d54ce1b545235ebf8040fb" candidate_uuid: "8cb0bbb22d814658852b0237922bbfb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f3f5a7d5afd41ceb534f34d36810852" is_pre_election: true
I20260619 16:43:21.631153 2445 raft_consensus.cc:2454] T e65e2817c4d54ce1b545235ebf8040fb P 4c9be31194f04e75874e50b6af75e143 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8cb0bbb22d814658852b0237922bbfb3 in term 0.
I20260619 16:43:21.631182 2301 raft_consensus.cc:2454] T e65e2817c4d54ce1b545235ebf8040fb P 0f3f5a7d5afd41ceb534f34d36810852 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8cb0bbb22d814658852b0237922bbfb3 in term 0.
I20260619 16:43:21.631562 2511 leader_election.cc:302] T e65e2817c4d54ce1b545235ebf8040fb P 8cb0bbb22d814658852b0237922bbfb3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c9be31194f04e75874e50b6af75e143, 8cb0bbb22d814658852b0237922bbfb3; no voters:
I20260619 16:43:21.631848 2655 raft_consensus.cc:2781] T e65e2817c4d54ce1b545235ebf8040fb P 8cb0bbb22d814658852b0237922bbfb3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 16:43:21.631943 2655 raft_consensus.cc:482] T e65e2817c4d54ce1b545235ebf8040fb P 8cb0bbb22d814658852b0237922bbfb3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 16:43:21.632053 2655 raft_consensus.cc:3037] T e65e2817c4d54ce1b545235ebf8040fb P 8cb0bbb22d814658852b0237922bbfb3 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:43:21.634264 2655 raft_consensus.cc:504] T e65e2817c4d54ce1b545235ebf8040fb P 8cb0bbb22d814658852b0237922bbfb3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f3f5a7d5afd41ceb534f34d36810852" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35421 } } peers { permanent_uuid: "4c9be31194f04e75874e50b6af75e143" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41989 } } peers { permanent_uuid: "8cb0bbb22d814658852b0237922bbfb3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45671 } }
I20260619 16:43:21.635047 2655 leader_election.cc:288] T e65e2817c4d54ce1b545235ebf8040fb P 8cb0bbb22d814658852b0237922bbfb3 [CANDIDATE]: Term 1 election: Requested vote from peers 0f3f5a7d5afd41ceb534f34d36810852 (127.0.97.1:35421), 4c9be31194f04e75874e50b6af75e143 (127.0.97.2:41989)
I20260619 16:43:21.635519 2301 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e65e2817c4d54ce1b545235ebf8040fb" candidate_uuid: "8cb0bbb22d814658852b0237922bbfb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f3f5a7d5afd41ceb534f34d36810852"
I20260619 16:43:21.635530 2445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e65e2817c4d54ce1b545235ebf8040fb" candidate_uuid: "8cb0bbb22d814658852b0237922bbfb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c9be31194f04e75874e50b6af75e143"
I20260619 16:43:21.635722 2301 raft_consensus.cc:3037] T e65e2817c4d54ce1b545235ebf8040fb P 0f3f5a7d5afd41ceb534f34d36810852 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:43:21.635742 2445 raft_consensus.cc:3037] T e65e2817c4d54ce1b545235ebf8040fb P 4c9be31194f04e75874e50b6af75e143 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:43:21.637851 2301 raft_consensus.cc:2454] T e65e2817c4d54ce1b545235ebf8040fb P 0f3f5a7d5afd41ceb534f34d36810852 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8cb0bbb22d814658852b0237922bbfb3 in term 1.
I20260619 16:43:21.638453 2510 leader_election.cc:302] T e65e2817c4d54ce1b545235ebf8040fb P 8cb0bbb22d814658852b0237922bbfb3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f3f5a7d5afd41ceb534f34d36810852, 8cb0bbb22d814658852b0237922bbfb3; no voters:
I20260619 16:43:21.638605 2445 raft_consensus.cc:2454] T e65e2817c4d54ce1b545235ebf8040fb P 4c9be31194f04e75874e50b6af75e143 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8cb0bbb22d814658852b0237922bbfb3 in term 1.
I20260619 16:43:21.638708 2655 raft_consensus.cc:2781] T e65e2817c4d54ce1b545235ebf8040fb P 8cb0bbb22d814658852b0237922bbfb3 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:43:21.638881 2655 raft_consensus.cc:686] T e65e2817c4d54ce1b545235ebf8040fb P 8cb0bbb22d814658852b0237922bbfb3 [term 1 LEADER]: Becoming Leader. State: Replica: 8cb0bbb22d814658852b0237922bbfb3, State: Running, Role: LEADER
I20260619 16:43:21.639140 2655 consensus_queue.cc:237] T e65e2817c4d54ce1b545235ebf8040fb P 8cb0bbb22d814658852b0237922bbfb3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f3f5a7d5afd41ceb534f34d36810852" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35421 } } peers { permanent_uuid: "4c9be31194f04e75874e50b6af75e143" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41989 } } peers { permanent_uuid: "8cb0bbb22d814658852b0237922bbfb3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45671 } }
I20260619 16:43:21.640674 2658 tablet_replica.cc:439] T e65e2817c4d54ce1b545235ebf8040fb P 8cb0bbb22d814658852b0237922bbfb3: TxnStatusTablet state changed. Reason: New leader 8cb0bbb22d814658852b0237922bbfb3. Latest consensus state: current_term: 1 leader_uuid: "8cb0bbb22d814658852b0237922bbfb3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f3f5a7d5afd41ceb534f34d36810852" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35421 } } peers { permanent_uuid: "4c9be31194f04e75874e50b6af75e143" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41989 } } peers { permanent_uuid: "8cb0bbb22d814658852b0237922bbfb3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45671 } } }
I20260619 16:43:21.640954 2658 tablet_replica.cc:442] T e65e2817c4d54ce1b545235ebf8040fb P 8cb0bbb22d814658852b0237922bbfb3: This TxnStatusTablet replica's current role is: LEADER
I20260619 16:43:21.641496 2670 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260619 16:43:21.641811 2670 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260619 16:43:21.642061 2144 catalog_manager.cc:5495] T e65e2817c4d54ce1b545235ebf8040fb P 8cb0bbb22d814658852b0237922bbfb3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8cb0bbb22d814658852b0237922bbfb3 (127.0.97.3). New cstate: current_term: 1 leader_uuid: "8cb0bbb22d814658852b0237922bbfb3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f3f5a7d5afd41ceb534f34d36810852" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35421 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4c9be31194f04e75874e50b6af75e143" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41989 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8cb0bbb22d814658852b0237922bbfb3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45671 } health_report { overall_health: HEALTHY } } }
I20260619 16:43:21.698514 2643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:21.693278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32980 (local address 127.0.97.3:45671)
0619 16:43:21.693448 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0619 16:43:21.693454 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:43:21.693604 (+ 150us) server_negotiation.cc:407] Connection header received
0619 16:43:21.693874 (+ 270us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:43:21.693878 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:43:21.693966 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:43:21.694091 (+ 125us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 16:43:21.694422 (+ 331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:21.695168 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:21.696395 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:21.696822 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:21.696940 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:43:21.697346 (+ 406us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 16:43:21.697928 (+ 582us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 16:43:21.698048 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0619 16:43:21.698140 (+ 92us) server_negotiation.cc:299] Negotiation successful
0619 16:43:21.698217 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260619 16:43:21.698562 2665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:21.693255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:45671 (local address 127.0.0.1:32980)
0619 16:43:21.693520 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0619 16:43:21.693535 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 16:43:21.693676 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:43:21.694110 (+ 434us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:43:21.694114 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:43:21.694126 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 16:43:21.694292 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:43:21.694300 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:43:21.695285 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:43:21.695291 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:43:21.696265 (+ 974us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:43:21.696277 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:43:21.696956 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:43:21.696962 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:43:21.697122 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:43:21.697171 (+ 49us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 16:43:21.698023 (+ 852us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 16:43:21.698043 (+ 20us) client_negotiation.cc:769] Sending connection context
0619 16:43:21.698118 (+ 75us) client_negotiation.cc:240] Negotiation successful
0619 16:43:21.698266 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54}
I20260619 16:43:21.706176 2301 raft_consensus.cc:1261] T e65e2817c4d54ce1b545235ebf8040fb P 0f3f5a7d5afd41ceb534f34d36810852 [term 1 FOLLOWER]: Refusing update from remote peer 8cb0bbb22d814658852b0237922bbfb3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:43:21.706189 2445 raft_consensus.cc:1261] T e65e2817c4d54ce1b545235ebf8040fb P 4c9be31194f04e75874e50b6af75e143 [term 1 FOLLOWER]: Refusing update from remote peer 8cb0bbb22d814658852b0237922bbfb3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:43:21.707149 2655 consensus_queue.cc:1035] T e65e2817c4d54ce1b545235ebf8040fb P 8cb0bbb22d814658852b0237922bbfb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c9be31194f04e75874e50b6af75e143" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:43:21.707477 2658 consensus_queue.cc:1035] T e65e2817c4d54ce1b545235ebf8040fb P 8cb0bbb22d814658852b0237922bbfb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f3f5a7d5afd41ceb534f34d36810852" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:43:21.717708 2654 tablet_replica.cc:439] T e65e2817c4d54ce1b545235ebf8040fb P 4c9be31194f04e75874e50b6af75e143: TxnStatusTablet state changed. Reason: New leader 8cb0bbb22d814658852b0237922bbfb3. Latest consensus state: current_term: 1 leader_uuid: "8cb0bbb22d814658852b0237922bbfb3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f3f5a7d5afd41ceb534f34d36810852" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35421 } } peers { permanent_uuid: "4c9be31194f04e75874e50b6af75e143" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41989 } } peers { permanent_uuid: "8cb0bbb22d814658852b0237922bbfb3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45671 } } }
I20260619 16:43:21.718058 2654 tablet_replica.cc:442] T e65e2817c4d54ce1b545235ebf8040fb P 4c9be31194f04e75874e50b6af75e143: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 16:43:21.720965 2673 mvcc.cc:205] Tried to move back new op lower bound from 7298610797378232320 to 7298610797116293120. Current Snapshot: MvccSnapshot[applied={T|T < 7298610797378232320}]
I20260619 16:43:21.722389 2656 tablet_replica.cc:439] T e65e2817c4d54ce1b545235ebf8040fb P 0f3f5a7d5afd41ceb534f34d36810852: TxnStatusTablet state changed. Reason: New leader 8cb0bbb22d814658852b0237922bbfb3. Latest consensus state: current_term: 1 leader_uuid: "8cb0bbb22d814658852b0237922bbfb3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f3f5a7d5afd41ceb534f34d36810852" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35421 } } peers { permanent_uuid: "4c9be31194f04e75874e50b6af75e143" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41989 } } peers { permanent_uuid: "8cb0bbb22d814658852b0237922bbfb3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45671 } } }
I20260619 16:43:21.722656 2656 tablet_replica.cc:442] T e65e2817c4d54ce1b545235ebf8040fb P 0f3f5a7d5afd41ceb534f34d36810852: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 16:43:21.729554 2675 mvcc.cc:205] Tried to move back new op lower bound from 7298610797378232320 to 7298610797116293120. Current Snapshot: MvccSnapshot[applied={T|T < 7298610797378232320}]
I20260619 16:43:21.731103 2654 tablet_replica.cc:439] T e65e2817c4d54ce1b545235ebf8040fb P 4c9be31194f04e75874e50b6af75e143: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8cb0bbb22d814658852b0237922bbfb3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f3f5a7d5afd41ceb534f34d36810852" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35421 } } peers { permanent_uuid: "4c9be31194f04e75874e50b6af75e143" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41989 } } peers { permanent_uuid: "8cb0bbb22d814658852b0237922bbfb3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45671 } } }
I20260619 16:43:21.731385 2654 tablet_replica.cc:442] T e65e2817c4d54ce1b545235ebf8040fb P 4c9be31194f04e75874e50b6af75e143: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 16:43:21.733706 2658 tablet_replica.cc:439] T e65e2817c4d54ce1b545235ebf8040fb P 8cb0bbb22d814658852b0237922bbfb3: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8cb0bbb22d814658852b0237922bbfb3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f3f5a7d5afd41ceb534f34d36810852" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35421 } } peers { permanent_uuid: "4c9be31194f04e75874e50b6af75e143" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41989 } } peers { permanent_uuid: "8cb0bbb22d814658852b0237922bbfb3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45671 } } }
I20260619 16:43:21.733974 2658 tablet_replica.cc:442] T e65e2817c4d54ce1b545235ebf8040fb P 8cb0bbb22d814658852b0237922bbfb3: This TxnStatusTablet replica's current role is: LEADER
I20260619 16:43:21.742744 2658 tablet_replica.cc:439] T e65e2817c4d54ce1b545235ebf8040fb P 8cb0bbb22d814658852b0237922bbfb3: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8cb0bbb22d814658852b0237922bbfb3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f3f5a7d5afd41ceb534f34d36810852" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35421 } } peers { permanent_uuid: "4c9be31194f04e75874e50b6af75e143" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41989 } } peers { permanent_uuid: "8cb0bbb22d814658852b0237922bbfb3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45671 } } }
I20260619 16:43:21.743031 2658 tablet_replica.cc:442] T e65e2817c4d54ce1b545235ebf8040fb P 8cb0bbb22d814658852b0237922bbfb3: This TxnStatusTablet replica's current role is: LEADER
I20260619 16:43:21.742799 2656 tablet_replica.cc:439] T e65e2817c4d54ce1b545235ebf8040fb P 0f3f5a7d5afd41ceb534f34d36810852: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8cb0bbb22d814658852b0237922bbfb3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f3f5a7d5afd41ceb534f34d36810852" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35421 } } peers { permanent_uuid: "4c9be31194f04e75874e50b6af75e143" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41989 } } peers { permanent_uuid: "8cb0bbb22d814658852b0237922bbfb3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45671 } } }
I20260619 16:43:21.745106 2656 tablet_replica.cc:442] T e65e2817c4d54ce1b545235ebf8040fb P 0f3f5a7d5afd41ceb534f34d36810852: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 16:43:21.760957 2643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:21.755161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32984 (local address 127.0.97.3:45671)
0619 16:43:21.755354 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0619 16:43:21.755361 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:43:21.755834 (+ 473us) server_negotiation.cc:407] Connection header received
0619 16:43:21.756020 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:43:21.756026 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:43:21.756119 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:43:21.756229 (+ 110us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 16:43:21.756605 (+ 376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:21.757836 (+ 1231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:21.759168 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:21.759780 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:21.759925 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:43:21.760266 (+ 341us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 16:43:21.760401 (+ 135us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 16:43:21.760486 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0619 16:43:21.760560 (+ 74us) server_negotiation.cc:299] Negotiation successful
0619 16:43:21.760635 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260619 16:43:21.777241 2643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:21.771317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:39711 (local address 127.0.97.3:45671)
0619 16:43:21.771493 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0619 16:43:21.771499 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:43:21.772010 (+ 511us) server_negotiation.cc:407] Connection header received
0619 16:43:21.772153 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:43:21.772158 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:43:21.772243 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:43:21.772331 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 16:43:21.772842 (+ 511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:21.773925 (+ 1083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:21.775356 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:21.776030 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:21.776147 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:43:21.776540 (+ 393us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 16:43:21.776675 (+ 135us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 16:43:21.776766 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0619 16:43:21.776860 (+ 94us) server_negotiation.cc:299] Negotiation successful
0619 16:43:21.776931 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":21,"server-negotiator.queue_time_us":50}
I20260619 16:43:21.777261 2686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:21.771327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:45671 (local address 127.0.97.3:39711)
0619 16:43:21.771927 (+ 600us) negotiation.cc:107] Waiting for socket to connect
0619 16:43:21.771942 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 16:43:21.772052 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:43:21.772344 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:43:21.772347 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:43:21.772357 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 16:43:21.772705 (+ 348us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:43:21.772712 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:43:21.774041 (+ 1329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:43:21.774045 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:43:21.775235 (+ 1190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:43:21.775245 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:43:21.776185 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:43:21.776190 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:43:21.776297 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:43:21.776345 (+ 48us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 16:43:21.776761 (+ 416us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 16:43:21.776768 (+ 7us) client_negotiation.cc:769] Sending connection context
0619 16:43:21.776821 (+ 53us) client_negotiation.cc:240] Negotiation successful
0619 16:43:21.776913 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":449,"mutex_wait_us":1,"thread_start_us":307,"threads_started":1}
I20260619 16:43:21.786159 2300 raft_consensus.cc:1261] T b0bce0c710c54d96af5f8113c8ab6a33 P 0f3f5a7d5afd41ceb534f34d36810852 [term 1 FOLLOWER]: Refusing update from remote peer 8cb0bbb22d814658852b0237922bbfb3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:43:21.786207 2445 raft_consensus.cc:1261] T b0bce0c710c54d96af5f8113c8ab6a33 P 4c9be31194f04e75874e50b6af75e143 [term 1 FOLLOWER]: Refusing update from remote peer 8cb0bbb22d814658852b0237922bbfb3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:43:21.786624 2658 consensus_queue.cc:1035] T b0bce0c710c54d96af5f8113c8ab6a33 P 8cb0bbb22d814658852b0237922bbfb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f3f5a7d5afd41ceb534f34d36810852" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:43:21.786880 2681 consensus_queue.cc:1035] T b0bce0c710c54d96af5f8113c8ab6a33 P 8cb0bbb22d814658852b0237922bbfb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c9be31194f04e75874e50b6af75e143" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:43:21.848732 2144 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59514:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:43:21.848974 2144 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59514:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:43:21.852197 2144 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 58e545bbf9084b46bd00d2e25bf47bb3: Sending DeleteTablet for 3 replicas of tablet b0bce0c710c54d96af5f8113c8ab6a33
I20260619 16:43:21.853161 2559 tablet_service.cc:1514] Processing DeleteTablet for tablet b0bce0c710c54d96af5f8113c8ab6a33 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:43:21 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:32966
I20260619 16:43:21.853353 2420 tablet_service.cc:1514] Processing DeleteTablet for tablet b0bce0c710c54d96af5f8113c8ab6a33 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:43:21 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:33140
I20260619 16:43:21.853365 2281 tablet_service.cc:1514] Processing DeleteTablet for tablet b0bce0c710c54d96af5f8113c8ab6a33 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:43:21 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:60222
I20260619 16:43:21.858605 2145 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59514:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 16:43:21.859261 2145 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 16:43:21.864264 2692 tablet_replica.cc:330] T b0bce0c710c54d96af5f8113c8ab6a33 P 0f3f5a7d5afd41ceb534f34d36810852: stopping tablet replica
I20260619 16:43:21.865216 2692 raft_consensus.cc:2229] T b0bce0c710c54d96af5f8113c8ab6a33 P 0f3f5a7d5afd41ceb534f34d36810852 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 16:43:21.866158 2692 raft_consensus.cc:2258] T b0bce0c710c54d96af5f8113c8ab6a33 P 0f3f5a7d5afd41ceb534f34d36810852 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:43:21.867756 2280 tablet_service.cc:1467] Processing CreateTablet for tablet ad712ff5f1b24dbba845fb661c8a3b4d (DEFAULT_TABLE table=test-table [id=f1ca5abb9c8340a58774a305b7ad10c3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:43:21.868077 2558 tablet_service.cc:1467] Processing CreateTablet for tablet ad712ff5f1b24dbba845fb661c8a3b4d (DEFAULT_TABLE table=test-table [id=f1ca5abb9c8340a58774a305b7ad10c3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:43:21.868199 2280 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ad712ff5f1b24dbba845fb661c8a3b4d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:43:21.868518 2558 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ad712ff5f1b24dbba845fb661c8a3b4d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:43:21.870296 2691 tablet_replica.cc:330] T b0bce0c710c54d96af5f8113c8ab6a33 P 8cb0bbb22d814658852b0237922bbfb3: stopping tablet replica
I20260619 16:43:21.870728 2691 raft_consensus.cc:2229] T b0bce0c710c54d96af5f8113c8ab6a33 P 8cb0bbb22d814658852b0237922bbfb3 [term 1 LEADER]: Raft consensus shutting down.
I20260619 16:43:21.872401 2691 raft_consensus.cc:2258] T b0bce0c710c54d96af5f8113c8ab6a33 P 8cb0bbb22d814658852b0237922bbfb3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:43:21.874217 2419 tablet_service.cc:1467] Processing CreateTablet for tablet ad712ff5f1b24dbba845fb661c8a3b4d (DEFAULT_TABLE table=test-table [id=f1ca5abb9c8340a58774a305b7ad10c3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:43:21.874693 2419 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ad712ff5f1b24dbba845fb661c8a3b4d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:43:21.875052 2648 tablet_bootstrap.cc:492] T ad712ff5f1b24dbba845fb661c8a3b4d P 8cb0bbb22d814658852b0237922bbfb3: Bootstrap starting.
I20260619 16:43:21.875360 2691 ts_tablet_manager.cc:1905] T b0bce0c710c54d96af5f8113c8ab6a33 P 8cb0bbb22d814658852b0237922bbfb3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:43:21.876744 2648 tablet_bootstrap.cc:654] T ad712ff5f1b24dbba845fb661c8a3b4d P 8cb0bbb22d814658852b0237922bbfb3: Neither blocks nor log segments found. Creating new log.
I20260619 16:43:21.881615 2650 tablet_bootstrap.cc:492] T ad712ff5f1b24dbba845fb661c8a3b4d P 0f3f5a7d5afd41ceb534f34d36810852: Bootstrap starting.
I20260619 16:43:21.881906 2649 tablet_bootstrap.cc:492] T ad712ff5f1b24dbba845fb661c8a3b4d P 4c9be31194f04e75874e50b6af75e143: Bootstrap starting.
I20260619 16:43:21.883133 2650 tablet_bootstrap.cc:654] T ad712ff5f1b24dbba845fb661c8a3b4d P 0f3f5a7d5afd41ceb534f34d36810852: Neither blocks nor log segments found. Creating new log.
I20260619 16:43:21.883369 2649 tablet_bootstrap.cc:654] T ad712ff5f1b24dbba845fb661c8a3b4d P 4c9be31194f04e75874e50b6af75e143: Neither blocks nor log segments found. Creating new log.
I20260619 16:43:21.885874 2691 ts_tablet_manager.cc:1918] T b0bce0c710c54d96af5f8113c8ab6a33 P 8cb0bbb22d814658852b0237922bbfb3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 16:43:21.886051 2691 log.cc:1196] T b0bce0c710c54d96af5f8113c8ab6a33 P 8cb0bbb22d814658852b0237922bbfb3: Deleting WAL directory at /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887385903558-388-0/minicluster-data/ts-2/wal/wals/b0bce0c710c54d96af5f8113c8ab6a33
I20260619 16:43:21.886607 2691 ts_tablet_manager.cc:1939] T b0bce0c710c54d96af5f8113c8ab6a33 P 8cb0bbb22d814658852b0237922bbfb3: Deleting consensus metadata
I20260619 16:43:21.888895 2129 catalog_manager.cc:4841] TS 8cb0bbb22d814658852b0237922bbfb3 (127.0.97.3:45671): tablet b0bce0c710c54d96af5f8113c8ab6a33 (table test-table [id=2aa280e7c7e44871a863c85bbcbb00b6]) successfully deleted
I20260619 16:43:21.892428 2649 tablet_bootstrap.cc:492] T ad712ff5f1b24dbba845fb661c8a3b4d P 4c9be31194f04e75874e50b6af75e143: No bootstrap required, opened a new log
I20260619 16:43:21.892688 2649 ts_tablet_manager.cc:1397] T ad712ff5f1b24dbba845fb661c8a3b4d P 4c9be31194f04e75874e50b6af75e143: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.002s
I20260619 16:43:21.893358 2692 ts_tablet_manager.cc:1905] T b0bce0c710c54d96af5f8113c8ab6a33 P 0f3f5a7d5afd41ceb534f34d36810852: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:43:21.893599 2649 raft_consensus.cc:348] T ad712ff5f1b24dbba845fb661c8a3b4d P 4c9be31194f04e75874e50b6af75e143 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f3f5a7d5afd41ceb534f34d36810852" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35421 } } peers { permanent_uuid: "4c9be31194f04e75874e50b6af75e143" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41989 } } peers { permanent_uuid: "8cb0bbb22d814658852b0237922bbfb3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45671 } }
I20260619 16:43:21.893971 2693 tablet_replica.cc:330] T b0bce0c710c54d96af5f8113c8ab6a33 P 4c9be31194f04e75874e50b6af75e143: stopping tablet replica
I20260619 16:43:21.894101 2649 raft_consensus.cc:374] T ad712ff5f1b24dbba845fb661c8a3b4d P 4c9be31194f04e75874e50b6af75e143 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:43:21.894229 2649 raft_consensus.cc:729] T ad712ff5f1b24dbba845fb661c8a3b4d P 4c9be31194f04e75874e50b6af75e143 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c9be31194f04e75874e50b6af75e143, State: Initialized, Role: FOLLOWER
I20260619 16:43:21.894423 2693 raft_consensus.cc:2229] T b0bce0c710c54d96af5f8113c8ab6a33 P 4c9be31194f04e75874e50b6af75e143 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 16:43:21.894773 2693 raft_consensus.cc:2258] T b0bce0c710c54d96af5f8113c8ab6a33 P 4c9be31194f04e75874e50b6af75e143 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:43:21.894662 2649 consensus_queue.cc:260] T ad712ff5f1b24dbba845fb661c8a3b4d P 4c9be31194f04e75874e50b6af75e143 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f3f5a7d5afd41ceb534f34d36810852" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35421 } } peers { permanent_uuid: "4c9be31194f04e75874e50b6af75e143" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41989 } } peers { permanent_uuid: "8cb0bbb22d814658852b0237922bbfb3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45671 } }
I20260619 16:43:21.896067 2649 ts_tablet_manager.cc:1428] T ad712ff5f1b24dbba845fb661c8a3b4d P 4c9be31194f04e75874e50b6af75e143: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260619 16:43:21.898490 2650 tablet_bootstrap.cc:492] T ad712ff5f1b24dbba845fb661c8a3b4d P 0f3f5a7d5afd41ceb534f34d36810852: No bootstrap required, opened a new log
I20260619 16:43:21.898729 2650 ts_tablet_manager.cc:1397] T ad712ff5f1b24dbba845fb661c8a3b4d P 0f3f5a7d5afd41ceb534f34d36810852: Time spent bootstrapping tablet: real 0.017s user 0.008s sys 0.000s
I20260619 16:43:21.899537 2650 raft_consensus.cc:348] T ad712ff5f1b24dbba845fb661c8a3b4d P 0f3f5a7d5afd41ceb534f34d36810852 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f3f5a7d5afd41ceb534f34d36810852" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35421 } } peers { permanent_uuid: "4c9be31194f04e75874e50b6af75e143" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41989 } } peers { permanent_uuid: "8cb0bbb22d814658852b0237922bbfb3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45671 } }
I20260619 16:43:21.899781 2650 raft_consensus.cc:374] T ad712ff5f1b24dbba845fb661c8a3b4d P 0f3f5a7d5afd41ceb534f34d36810852 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:43:21.899838 2693 ts_tablet_manager.cc:1905] T b0bce0c710c54d96af5f8113c8ab6a33 P 4c9be31194f04e75874e50b6af75e143: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:43:21.899858 2650 raft_consensus.cc:729] T ad712ff5f1b24dbba845fb661c8a3b4d P 0f3f5a7d5afd41ceb534f34d36810852 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f3f5a7d5afd41ceb534f34d36810852, State: Initialized, Role: FOLLOWER
I20260619 16:43:21.900089 2650 consensus_queue.cc:260] T ad712ff5f1b24dbba845fb661c8a3b4d P 0f3f5a7d5afd41ceb534f34d36810852 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f3f5a7d5afd41ceb534f34d36810852" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35421 } } peers { permanent_uuid: "4c9be31194f04e75874e50b6af75e143" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41989 } } peers { permanent_uuid: "8cb0bbb22d814658852b0237922bbfb3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45671 } }
I20260619 16:43:21.900660 2650 ts_tablet_manager.cc:1428] T ad712ff5f1b24dbba845fb661c8a3b4d P 0f3f5a7d5afd41ceb534f34d36810852: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260619 16:43:21.902067 2648 tablet_bootstrap.cc:492] T ad712ff5f1b24dbba845fb661c8a3b4d P 8cb0bbb22d814658852b0237922bbfb3: No bootstrap required, opened a new log
I20260619 16:43:21.902303 2648 ts_tablet_manager.cc:1397] T ad712ff5f1b24dbba845fb661c8a3b4d P 8cb0bbb22d814658852b0237922bbfb3: Time spent bootstrapping tablet: real 0.027s user 0.003s sys 0.011s
I20260619 16:43:21.903162 2648 raft_consensus.cc:348] T ad712ff5f1b24dbba845fb661c8a3b4d P 8cb0bbb22d814658852b0237922bbfb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f3f5a7d5afd41ceb534f34d36810852" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35421 } } peers { permanent_uuid: "4c9be31194f04e75874e50b6af75e143" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41989 } } peers { permanent_uuid: "8cb0bbb22d814658852b0237922bbfb3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45671 } }
I20260619 16:43:21.903469 2648 raft_consensus.cc:374] T ad712ff5f1b24dbba845fb661c8a3b4d P 8cb0bbb22d814658852b0237922bbfb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:43:21.903589 2648 raft_consensus.cc:729] T ad712ff5f1b24dbba845fb661c8a3b4d P 8cb0bbb22d814658852b0237922bbfb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8cb0bbb22d814658852b0237922bbfb3, State: Initialized, Role: FOLLOWER
I20260619 16:43:21.903899 2648 consensus_queue.cc:260] T ad712ff5f1b24dbba845fb661c8a3b4d P 8cb0bbb22d814658852b0237922bbfb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f3f5a7d5afd41ceb534f34d36810852" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35421 } } peers { permanent_uuid: "4c9be31194f04e75874e50b6af75e143" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41989 } } peers { permanent_uuid: "8cb0bbb22d814658852b0237922bbfb3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45671 } }
I20260619 16:43:21.906701 2648 ts_tablet_manager.cc:1428] T ad712ff5f1b24dbba845fb661c8a3b4d P 8cb0bbb22d814658852b0237922bbfb3: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260619 16:43:21.908418 2692 ts_tablet_manager.cc:1918] T b0bce0c710c54d96af5f8113c8ab6a33 P 0f3f5a7d5afd41ceb534f34d36810852: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 16:43:21.908618 2692 log.cc:1196] T b0bce0c710c54d96af5f8113c8ab6a33 P 0f3f5a7d5afd41ceb534f34d36810852: Deleting WAL directory at /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887385903558-388-0/minicluster-data/ts-0/wal/wals/b0bce0c710c54d96af5f8113c8ab6a33
I20260619 16:43:21.909135 2692 ts_tablet_manager.cc:1939] T b0bce0c710c54d96af5f8113c8ab6a33 P 0f3f5a7d5afd41ceb534f34d36810852: Deleting consensus metadata
I20260619 16:43:21.910872 2129 catalog_manager.cc:4841] TS 0f3f5a7d5afd41ceb534f34d36810852 (127.0.97.1:35421): tablet b0bce0c710c54d96af5f8113c8ab6a33 (table test-table [id=2aa280e7c7e44871a863c85bbcbb00b6]) successfully deleted
I20260619 16:43:21.911070 2693 ts_tablet_manager.cc:1918] T b0bce0c710c54d96af5f8113c8ab6a33 P 4c9be31194f04e75874e50b6af75e143: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 16:43:21.911194 2693 log.cc:1196] T b0bce0c710c54d96af5f8113c8ab6a33 P 4c9be31194f04e75874e50b6af75e143: Deleting WAL directory at /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887385903558-388-0/minicluster-data/ts-1/wal/wals/b0bce0c710c54d96af5f8113c8ab6a33
I20260619 16:43:21.911569 2693 ts_tablet_manager.cc:1939] T b0bce0c710c54d96af5f8113c8ab6a33 P 4c9be31194f04e75874e50b6af75e143: Deleting consensus metadata
I20260619 16:43:21.912910 2130 catalog_manager.cc:4841] TS 4c9be31194f04e75874e50b6af75e143 (127.0.97.2:41989): tablet b0bce0c710c54d96af5f8113c8ab6a33 (table test-table [id=2aa280e7c7e44871a863c85bbcbb00b6]) successfully deleted
I20260619 16:43:22.123842 2655 raft_consensus.cc:482] T ad712ff5f1b24dbba845fb661c8a3b4d P 8cb0bbb22d814658852b0237922bbfb3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 16:43:22.124070 2655 raft_consensus.cc:504] T ad712ff5f1b24dbba845fb661c8a3b4d P 8cb0bbb22d814658852b0237922bbfb3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f3f5a7d5afd41ceb534f34d36810852" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35421 } } peers { permanent_uuid: "4c9be31194f04e75874e50b6af75e143" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41989 } } peers { permanent_uuid: "8cb0bbb22d814658852b0237922bbfb3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45671 } }
I20260619 16:43:22.124812 2655 leader_election.cc:288] T ad712ff5f1b24dbba845fb661c8a3b4d P 8cb0bbb22d814658852b0237922bbfb3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0f3f5a7d5afd41ceb534f34d36810852 (127.0.97.1:35421), 4c9be31194f04e75874e50b6af75e143 (127.0.97.2:41989)
I20260619 16:43:22.125488 2300 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad712ff5f1b24dbba845fb661c8a3b4d" candidate_uuid: "8cb0bbb22d814658852b0237922bbfb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f3f5a7d5afd41ceb534f34d36810852" is_pre_election: true
I20260619 16:43:22.125535 2444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad712ff5f1b24dbba845fb661c8a3b4d" candidate_uuid: "8cb0bbb22d814658852b0237922bbfb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c9be31194f04e75874e50b6af75e143" is_pre_election: true
I20260619 16:43:22.125799 2444 raft_consensus.cc:2454] T ad712ff5f1b24dbba845fb661c8a3b4d P 4c9be31194f04e75874e50b6af75e143 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8cb0bbb22d814658852b0237922bbfb3 in term 0.
I20260619 16:43:22.125835 2300 raft_consensus.cc:2454] T ad712ff5f1b24dbba845fb661c8a3b4d P 0f3f5a7d5afd41ceb534f34d36810852 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8cb0bbb22d814658852b0237922bbfb3 in term 0.
I20260619 16:43:22.126230 2511 leader_election.cc:302] T ad712ff5f1b24dbba845fb661c8a3b4d P 8cb0bbb22d814658852b0237922bbfb3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c9be31194f04e75874e50b6af75e143, 8cb0bbb22d814658852b0237922bbfb3; no voters:
I20260619 16:43:22.126541 2655 raft_consensus.cc:2781] T ad712ff5f1b24dbba845fb661c8a3b4d P 8cb0bbb22d814658852b0237922bbfb3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 16:43:22.126641 2655 raft_consensus.cc:482] T ad712ff5f1b24dbba845fb661c8a3b4d P 8cb0bbb22d814658852b0237922bbfb3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 16:43:22.126739 2655 raft_consensus.cc:3037] T ad712ff5f1b24dbba845fb661c8a3b4d P 8cb0bbb22d814658852b0237922bbfb3 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:43:22.129799 2655 raft_consensus.cc:504] T ad712ff5f1b24dbba845fb661c8a3b4d P 8cb0bbb22d814658852b0237922bbfb3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f3f5a7d5afd41ceb534f34d36810852" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35421 } } peers { permanent_uuid: "4c9be31194f04e75874e50b6af75e143" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41989 } } peers { permanent_uuid: "8cb0bbb22d814658852b0237922bbfb3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45671 } }
I20260619 16:43:22.130990 2300 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad712ff5f1b24dbba845fb661c8a3b4d" candidate_uuid: "8cb0bbb22d814658852b0237922bbfb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f3f5a7d5afd41ceb534f34d36810852"
I20260619 16:43:22.131181 2300 raft_consensus.cc:3037] T ad712ff5f1b24dbba845fb661c8a3b4d P 0f3f5a7d5afd41ceb534f34d36810852 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:43:22.134086 2300 raft_consensus.cc:2454] T ad712ff5f1b24dbba845fb661c8a3b4d P 0f3f5a7d5afd41ceb534f34d36810852 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8cb0bbb22d814658852b0237922bbfb3 in term 1.
I20260619 16:43:22.134536 2510 leader_election.cc:302] T ad712ff5f1b24dbba845fb661c8a3b4d P 8cb0bbb22d814658852b0237922bbfb3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f3f5a7d5afd41ceb534f34d36810852, 8cb0bbb22d814658852b0237922bbfb3; no voters:
I20260619 16:43:22.134799 2690 raft_consensus.cc:2781] T ad712ff5f1b24dbba845fb661c8a3b4d P 8cb0bbb22d814658852b0237922bbfb3 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:43:22.134961 2690 raft_consensus.cc:686] T ad712ff5f1b24dbba845fb661c8a3b4d P 8cb0bbb22d814658852b0237922bbfb3 [term 1 LEADER]: Becoming Leader. State: Replica: 8cb0bbb22d814658852b0237922bbfb3, State: Running, Role: LEADER
I20260619 16:43:22.135298 2690 consensus_queue.cc:237] T ad712ff5f1b24dbba845fb661c8a3b4d P 8cb0bbb22d814658852b0237922bbfb3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f3f5a7d5afd41ceb534f34d36810852" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35421 } } peers { permanent_uuid: "4c9be31194f04e75874e50b6af75e143" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41989 } } peers { permanent_uuid: "8cb0bbb22d814658852b0237922bbfb3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45671 } }
I20260619 16:43:22.138433 2444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad712ff5f1b24dbba845fb661c8a3b4d" candidate_uuid: "8cb0bbb22d814658852b0237922bbfb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c9be31194f04e75874e50b6af75e143"
I20260619 16:43:22.138631 2444 raft_consensus.cc:3037] T ad712ff5f1b24dbba845fb661c8a3b4d P 4c9be31194f04e75874e50b6af75e143 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:43:22.141469 2444 raft_consensus.cc:2454] T ad712ff5f1b24dbba845fb661c8a3b4d P 4c9be31194f04e75874e50b6af75e143 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8cb0bbb22d814658852b0237922bbfb3 in term 1.
I20260619 16:43:22.143762 2144 catalog_manager.cc:5495] T ad712ff5f1b24dbba845fb661c8a3b4d P 8cb0bbb22d814658852b0237922bbfb3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8cb0bbb22d814658852b0237922bbfb3 (127.0.97.3). New cstate: current_term: 1 leader_uuid: "8cb0bbb22d814658852b0237922bbfb3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f3f5a7d5afd41ceb534f34d36810852" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35421 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4c9be31194f04e75874e50b6af75e143" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41989 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8cb0bbb22d814658852b0237922bbfb3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45671 } health_report { overall_health: HEALTHY } } }
I20260619 16:43:22.147473 2655 leader_election.cc:288] T ad712ff5f1b24dbba845fb661c8a3b4d P 8cb0bbb22d814658852b0237922bbfb3 [CANDIDATE]: Term 1 election: Requested vote from peers 0f3f5a7d5afd41ceb534f34d36810852 (127.0.97.1:35421), 4c9be31194f04e75874e50b6af75e143 (127.0.97.2:41989)
I20260619 16:43:22.186066 2445 raft_consensus.cc:1261] T ad712ff5f1b24dbba845fb661c8a3b4d P 4c9be31194f04e75874e50b6af75e143 [term 1 FOLLOWER]: Refusing update from remote peer 8cb0bbb22d814658852b0237922bbfb3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:43:22.186045 2300 raft_consensus.cc:1261] T ad712ff5f1b24dbba845fb661c8a3b4d P 0f3f5a7d5afd41ceb534f34d36810852 [term 1 FOLLOWER]: Refusing update from remote peer 8cb0bbb22d814658852b0237922bbfb3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:43:22.186532 2655 consensus_queue.cc:1035] T ad712ff5f1b24dbba845fb661c8a3b4d P 8cb0bbb22d814658852b0237922bbfb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f3f5a7d5afd41ceb534f34d36810852" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:43:22.186796 2690 consensus_queue.cc:1035] T ad712ff5f1b24dbba845fb661c8a3b4d P 8cb0bbb22d814658852b0237922bbfb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c9be31194f04e75874e50b6af75e143" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:43:22.192862 2674 mvcc.cc:205] Tried to move back new op lower bound from 7298610799348715520 to 7298610799148937216. Current Snapshot: MvccSnapshot[applied={T|T < 7298610799348715520}]
W20260619 16:43:22.210804 2673 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260619 16:43:23.784005 2144 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59514:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:43:23.784227 2144 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59514:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:43:23.788182 2144 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 58e545bbf9084b46bd00d2e25bf47bb3: Sending DeleteTablet for 3 replicas of tablet ad712ff5f1b24dbba845fb661c8a3b4d
I20260619 16:43:23.788959 2280 tablet_service.cc:1514] Processing DeleteTablet for tablet ad712ff5f1b24dbba845fb661c8a3b4d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:43:23 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:60222
I20260619 16:43:23.789011 2420 tablet_service.cc:1514] Processing DeleteTablet for tablet ad712ff5f1b24dbba845fb661c8a3b4d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:43:23 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:33140
I20260619 16:43:23.789065 2558 tablet_service.cc:1514] Processing DeleteTablet for tablet ad712ff5f1b24dbba845fb661c8a3b4d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:43:23 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:32966
I20260619 16:43:23.801402 2721 tablet_replica.cc:330] T ad712ff5f1b24dbba845fb661c8a3b4d P 0f3f5a7d5afd41ceb534f34d36810852: stopping tablet replica
I20260619 16:43:23.801635 2721 raft_consensus.cc:2229] T ad712ff5f1b24dbba845fb661c8a3b4d P 0f3f5a7d5afd41ceb534f34d36810852 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 16:43:23.801679 2722 tablet_replica.cc:330] T ad712ff5f1b24dbba845fb661c8a3b4d P 8cb0bbb22d814658852b0237922bbfb3: stopping tablet replica
I20260619 16:43:23.801996 2722 raft_consensus.cc:2229] T ad712ff5f1b24dbba845fb661c8a3b4d P 8cb0bbb22d814658852b0237922bbfb3 [term 1 LEADER]: Raft consensus shutting down.
I20260619 16:43:23.802397 2722 raft_consensus.cc:2258] T ad712ff5f1b24dbba845fb661c8a3b4d P 8cb0bbb22d814658852b0237922bbfb3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:43:23.802709 2721 raft_consensus.cc:2258] T ad712ff5f1b24dbba845fb661c8a3b4d P 0f3f5a7d5afd41ceb534f34d36810852 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:43:23.803426 2722 ts_tablet_manager.cc:1905] T ad712ff5f1b24dbba845fb661c8a3b4d P 8cb0bbb22d814658852b0237922bbfb3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:43:23.804049 2720 tablet_replica.cc:330] T ad712ff5f1b24dbba845fb661c8a3b4d P 4c9be31194f04e75874e50b6af75e143: stopping tablet replica
I20260619 16:43:23.804077 2721 ts_tablet_manager.cc:1905] T ad712ff5f1b24dbba845fb661c8a3b4d P 0f3f5a7d5afd41ceb534f34d36810852: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:43:23.804260 2720 raft_consensus.cc:2229] T ad712ff5f1b24dbba845fb661c8a3b4d P 4c9be31194f04e75874e50b6af75e143 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 16:43:23.804512 2720 raft_consensus.cc:2258] T ad712ff5f1b24dbba845fb661c8a3b4d P 4c9be31194f04e75874e50b6af75e143 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:43:23.805526 2720 ts_tablet_manager.cc:1905] T ad712ff5f1b24dbba845fb661c8a3b4d P 4c9be31194f04e75874e50b6af75e143: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:43:23.810524 2722 ts_tablet_manager.cc:1918] T ad712ff5f1b24dbba845fb661c8a3b4d P 8cb0bbb22d814658852b0237922bbfb3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260619 16:43:23.810734 2722 log.cc:1196] T ad712ff5f1b24dbba845fb661c8a3b4d P 8cb0bbb22d814658852b0237922bbfb3: Deleting WAL directory at /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887385903558-388-0/minicluster-data/ts-2/wal/wals/ad712ff5f1b24dbba845fb661c8a3b4d
I20260619 16:43:23.811153 2721 ts_tablet_manager.cc:1918] T ad712ff5f1b24dbba845fb661c8a3b4d P 0f3f5a7d5afd41ceb534f34d36810852: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260619 16:43:23.811291 2722 ts_tablet_manager.cc:1939] T ad712ff5f1b24dbba845fb661c8a3b4d P 8cb0bbb22d814658852b0237922bbfb3: Deleting consensus metadata
I20260619 16:43:23.811342 2721 log.cc:1196] T ad712ff5f1b24dbba845fb661c8a3b4d P 0f3f5a7d5afd41ceb534f34d36810852: Deleting WAL directory at /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887385903558-388-0/minicluster-data/ts-0/wal/wals/ad712ff5f1b24dbba845fb661c8a3b4d
I20260619 16:43:23.811544 2720 ts_tablet_manager.cc:1918] T ad712ff5f1b24dbba845fb661c8a3b4d P 4c9be31194f04e75874e50b6af75e143: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260619 16:43:23.811707 2720 log.cc:1196] T ad712ff5f1b24dbba845fb661c8a3b4d P 4c9be31194f04e75874e50b6af75e143: Deleting WAL directory at /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887385903558-388-0/minicluster-data/ts-1/wal/wals/ad712ff5f1b24dbba845fb661c8a3b4d
I20260619 16:43:23.811925 2721 ts_tablet_manager.cc:1939] T ad712ff5f1b24dbba845fb661c8a3b4d P 0f3f5a7d5afd41ceb534f34d36810852: Deleting consensus metadata
I20260619 16:43:23.812120 2720 ts_tablet_manager.cc:1939] T ad712ff5f1b24dbba845fb661c8a3b4d P 4c9be31194f04e75874e50b6af75e143: Deleting consensus metadata
I20260619 16:43:23.813055 2129 catalog_manager.cc:4841] TS 0f3f5a7d5afd41ceb534f34d36810852 (127.0.97.1:35421): tablet ad712ff5f1b24dbba845fb661c8a3b4d (table test-table [id=f1ca5abb9c8340a58774a305b7ad10c3]) successfully deleted
I20260619 16:43:23.813104 2130 catalog_manager.cc:4841] TS 4c9be31194f04e75874e50b6af75e143 (127.0.97.2:41989): tablet ad712ff5f1b24dbba845fb661c8a3b4d (table test-table [id=f1ca5abb9c8340a58774a305b7ad10c3]) successfully deleted
I20260619 16:43:23.813357 2129 catalog_manager.cc:4841] TS 8cb0bbb22d814658852b0237922bbfb3 (127.0.97.3:45671): tablet ad712ff5f1b24dbba845fb661c8a3b4d (table test-table [id=f1ca5abb9c8340a58774a305b7ad10c3]) successfully deleted
Jun 19 16:43:23 dist-test-slave-wvz7 krb5kdc[2100](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887403, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 19 16:43:23 dist-test-slave-wvz7 krb5kdc[2100](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887403, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 16:43:23.846598 2730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:23.829727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49432 (local address 127.0.97.62:46341)
0619 16:43:23.830281 (+ 554us) server_negotiation.cc:206] Beginning negotiation
0619 16:43:23.830286 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:43:23.830310 (+ 24us) server_negotiation.cc:407] Connection header received
0619 16:43:23.830463 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:43:23.830467 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:43:23.830526 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:43:23.830614 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:43:23.831848 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:23.832579 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:23.833810 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:23.834226 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:23.834329 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:43:23.838536 (+ 4207us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:43:23.838541 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:43:23.838544 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:43:23.838574 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:43:23.844237 (+ 5663us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:43:23.844848 (+ 611us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:43:23.844852 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:43:23.844854 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:43:23.844980 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:43:23.845304 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:43:23.845307 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:43:23.845309 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:43:23.845562 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:43:23.845800 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0619 16:43:23.846077 (+ 277us) server_negotiation.cc:299] Negotiation successful
0619 16:43:23.846281 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":410,"thread_start_us":261,"threads_started":1}
I20260619 16:43:23.849710 2144 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49432:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 16:43:23.850332 2144 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 16:43:23.856149 2420 tablet_service.cc:1467] Processing CreateTablet for tablet 15ab05270823492481cad281abdbb2f9 (DEFAULT_TABLE table=test-table [id=5415dc5964564d4da2869209da4705bd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:43:23.856114 2280 tablet_service.cc:1467] Processing CreateTablet for tablet 15ab05270823492481cad281abdbb2f9 (DEFAULT_TABLE table=test-table [id=5415dc5964564d4da2869209da4705bd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:43:23.856287 2558 tablet_service.cc:1467] Processing CreateTablet for tablet 15ab05270823492481cad281abdbb2f9 (DEFAULT_TABLE table=test-table [id=5415dc5964564d4da2869209da4705bd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:43:23.856540 2420 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 15ab05270823492481cad281abdbb2f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:43:23.856554 2280 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 15ab05270823492481cad281abdbb2f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:43:23.856698 2558 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 15ab05270823492481cad281abdbb2f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:43:23.861845 2731 tablet_bootstrap.cc:492] T 15ab05270823492481cad281abdbb2f9 P 8cb0bbb22d814658852b0237922bbfb3: Bootstrap starting.
I20260619 16:43:23.863224 2732 tablet_bootstrap.cc:492] T 15ab05270823492481cad281abdbb2f9 P 0f3f5a7d5afd41ceb534f34d36810852: Bootstrap starting.
I20260619 16:43:23.863602 2733 tablet_bootstrap.cc:492] T 15ab05270823492481cad281abdbb2f9 P 4c9be31194f04e75874e50b6af75e143: Bootstrap starting.
I20260619 16:43:23.863927 2731 tablet_bootstrap.cc:654] T 15ab05270823492481cad281abdbb2f9 P 8cb0bbb22d814658852b0237922bbfb3: Neither blocks nor log segments found. Creating new log.
I20260619 16:43:23.864746 2732 tablet_bootstrap.cc:654] T 15ab05270823492481cad281abdbb2f9 P 0f3f5a7d5afd41ceb534f34d36810852: Neither blocks nor log segments found. Creating new log.
I20260619 16:43:23.865011 2733 tablet_bootstrap.cc:654] T 15ab05270823492481cad281abdbb2f9 P 4c9be31194f04e75874e50b6af75e143: Neither blocks nor log segments found. Creating new log.
I20260619 16:43:23.867457 2732 tablet_bootstrap.cc:492] T 15ab05270823492481cad281abdbb2f9 P 0f3f5a7d5afd41ceb534f34d36810852: No bootstrap required, opened a new log
I20260619 16:43:23.867600 2732 ts_tablet_manager.cc:1397] T 15ab05270823492481cad281abdbb2f9 P 0f3f5a7d5afd41ceb534f34d36810852: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.002s
I20260619 16:43:23.868076 2733 tablet_bootstrap.cc:492] T 15ab05270823492481cad281abdbb2f9 P 4c9be31194f04e75874e50b6af75e143: No bootstrap required, opened a new log
I20260619 16:43:23.868208 2733 ts_tablet_manager.cc:1397] T 15ab05270823492481cad281abdbb2f9 P 4c9be31194f04e75874e50b6af75e143: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260619 16:43:23.868232 2731 tablet_bootstrap.cc:492] T 15ab05270823492481cad281abdbb2f9 P 8cb0bbb22d814658852b0237922bbfb3: No bootstrap required, opened a new log
I20260619 16:43:23.868400 2731 ts_tablet_manager.cc:1397] T 15ab05270823492481cad281abdbb2f9 P 8cb0bbb22d814658852b0237922bbfb3: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.003s
I20260619 16:43:23.868333 2732 raft_consensus.cc:348] T 15ab05270823492481cad281abdbb2f9 P 0f3f5a7d5afd41ceb534f34d36810852 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f3f5a7d5afd41ceb534f34d36810852" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35421 } } peers { permanent_uuid: "4c9be31194f04e75874e50b6af75e143" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41989 } } peers { permanent_uuid: "8cb0bbb22d814658852b0237922bbfb3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45671 } }
I20260619 16:43:23.868594 2732 raft_consensus.cc:374] T 15ab05270823492481cad281abdbb2f9 P 0f3f5a7d5afd41ceb534f34d36810852 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:43:23.868718 2732 raft_consensus.cc:729] T 15ab05270823492481cad281abdbb2f9 P 0f3f5a7d5afd41ceb534f34d36810852 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f3f5a7d5afd41ceb534f34d36810852, State: Initialized, Role: FOLLOWER
I20260619 16:43:23.868922 2733 raft_consensus.cc:348] T 15ab05270823492481cad281abdbb2f9 P 4c9be31194f04e75874e50b6af75e143 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f3f5a7d5afd41ceb534f34d36810852" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35421 } } peers { permanent_uuid: "4c9be31194f04e75874e50b6af75e143" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41989 } } peers { permanent_uuid: "8cb0bbb22d814658852b0237922bbfb3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45671 } }
I20260619 16:43:23.868991 2732 consensus_queue.cc:260] T 15ab05270823492481cad281abdbb2f9 P 0f3f5a7d5afd41ceb534f34d36810852 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f3f5a7d5afd41ceb534f34d36810852" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35421 } } peers { permanent_uuid: "4c9be31194f04e75874e50b6af75e143" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41989 } } peers { permanent_uuid: "8cb0bbb22d814658852b0237922bbfb3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45671 } }
I20260619 16:43:23.869161 2733 raft_consensus.cc:374] T 15ab05270823492481cad281abdbb2f9 P 4c9be31194f04e75874e50b6af75e143 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:43:23.869261 2733 raft_consensus.cc:729] T 15ab05270823492481cad281abdbb2f9 P 4c9be31194f04e75874e50b6af75e143 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c9be31194f04e75874e50b6af75e143, State: Initialized, Role: FOLLOWER
I20260619 16:43:23.869127 2731 raft_consensus.cc:348] T 15ab05270823492481cad281abdbb2f9 P 8cb0bbb22d814658852b0237922bbfb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f3f5a7d5afd41ceb534f34d36810852" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35421 } } peers { permanent_uuid: "4c9be31194f04e75874e50b6af75e143" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41989 } } peers { permanent_uuid: "8cb0bbb22d814658852b0237922bbfb3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45671 } }
I20260619 16:43:23.869351 2731 raft_consensus.cc:374] T 15ab05270823492481cad281abdbb2f9 P 8cb0bbb22d814658852b0237922bbfb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:43:23.869468 2731 raft_consensus.cc:729] T 15ab05270823492481cad281abdbb2f9 P 8cb0bbb22d814658852b0237922bbfb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8cb0bbb22d814658852b0237922bbfb3, State: Initialized, Role: FOLLOWER
I20260619 16:43:23.869534 2733 consensus_queue.cc:260] T 15ab05270823492481cad281abdbb2f9 P 4c9be31194f04e75874e50b6af75e143 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f3f5a7d5afd41ceb534f34d36810852" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35421 } } peers { permanent_uuid: "4c9be31194f04e75874e50b6af75e143" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41989 } } peers { permanent_uuid: "8cb0bbb22d814658852b0237922bbfb3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45671 } }
I20260619 16:43:23.869729 2731 consensus_queue.cc:260] T 15ab05270823492481cad281abdbb2f9 P 8cb0bbb22d814658852b0237922bbfb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f3f5a7d5afd41ceb534f34d36810852" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35421 } } peers { permanent_uuid: "4c9be31194f04e75874e50b6af75e143" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41989 } } peers { permanent_uuid: "8cb0bbb22d814658852b0237922bbfb3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45671 } }
I20260619 16:43:23.870417 2733 ts_tablet_manager.cc:1428] T 15ab05270823492481cad281abdbb2f9 P 4c9be31194f04e75874e50b6af75e143: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260619 16:43:23.871088 2732 ts_tablet_manager.cc:1428] T 15ab05270823492481cad281abdbb2f9 P 0f3f5a7d5afd41ceb534f34d36810852: Time spent starting tablet: real 0.003s user 0.000s sys 0.004s
I20260619 16:43:23.871701 2731 ts_tablet_manager.cc:1428] T 15ab05270823492481cad281abdbb2f9 P 8cb0bbb22d814658852b0237922bbfb3: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260619 16:43:24.048565 2715 raft_consensus.cc:482] T 15ab05270823492481cad281abdbb2f9 P 8cb0bbb22d814658852b0237922bbfb3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 16:43:24.048810 2715 raft_consensus.cc:504] T 15ab05270823492481cad281abdbb2f9 P 8cb0bbb22d814658852b0237922bbfb3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f3f5a7d5afd41ceb534f34d36810852" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35421 } } peers { permanent_uuid: "4c9be31194f04e75874e50b6af75e143" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41989 } } peers { permanent_uuid: "8cb0bbb22d814658852b0237922bbfb3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45671 } }
I20260619 16:43:24.049564 2715 leader_election.cc:288] T 15ab05270823492481cad281abdbb2f9 P 8cb0bbb22d814658852b0237922bbfb3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0f3f5a7d5afd41ceb534f34d36810852 (127.0.97.1:35421), 4c9be31194f04e75874e50b6af75e143 (127.0.97.2:41989)
I20260619 16:43:24.050057 2301 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15ab05270823492481cad281abdbb2f9" candidate_uuid: "8cb0bbb22d814658852b0237922bbfb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f3f5a7d5afd41ceb534f34d36810852" is_pre_election: true
I20260619 16:43:24.050130 2445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15ab05270823492481cad281abdbb2f9" candidate_uuid: "8cb0bbb22d814658852b0237922bbfb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c9be31194f04e75874e50b6af75e143" is_pre_election: true
I20260619 16:43:24.050288 2301 raft_consensus.cc:2454] T 15ab05270823492481cad281abdbb2f9 P 0f3f5a7d5afd41ceb534f34d36810852 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8cb0bbb22d814658852b0237922bbfb3 in term 0.
I20260619 16:43:24.050426 2445 raft_consensus.cc:2454] T 15ab05270823492481cad281abdbb2f9 P 4c9be31194f04e75874e50b6af75e143 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8cb0bbb22d814658852b0237922bbfb3 in term 0.
I20260619 16:43:24.050765 2510 leader_election.cc:302] T 15ab05270823492481cad281abdbb2f9 P 8cb0bbb22d814658852b0237922bbfb3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f3f5a7d5afd41ceb534f34d36810852, 8cb0bbb22d814658852b0237922bbfb3; no voters:
I20260619 16:43:24.051030 2715 raft_consensus.cc:2781] T 15ab05270823492481cad281abdbb2f9 P 8cb0bbb22d814658852b0237922bbfb3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 16:43:24.051142 2715 raft_consensus.cc:482] T 15ab05270823492481cad281abdbb2f9 P 8cb0bbb22d814658852b0237922bbfb3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 16:43:24.051229 2715 raft_consensus.cc:3037] T 15ab05270823492481cad281abdbb2f9 P 8cb0bbb22d814658852b0237922bbfb3 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:43:24.053153 2715 raft_consensus.cc:504] T 15ab05270823492481cad281abdbb2f9 P 8cb0bbb22d814658852b0237922bbfb3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f3f5a7d5afd41ceb534f34d36810852" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35421 } } peers { permanent_uuid: "4c9be31194f04e75874e50b6af75e143" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41989 } } peers { permanent_uuid: "8cb0bbb22d814658852b0237922bbfb3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45671 } }
I20260619 16:43:24.053668 2715 leader_election.cc:288] T 15ab05270823492481cad281abdbb2f9 P 8cb0bbb22d814658852b0237922bbfb3 [CANDIDATE]: Term 1 election: Requested vote from peers 0f3f5a7d5afd41ceb534f34d36810852 (127.0.97.1:35421), 4c9be31194f04e75874e50b6af75e143 (127.0.97.2:41989)
I20260619 16:43:24.054133 2301 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15ab05270823492481cad281abdbb2f9" candidate_uuid: "8cb0bbb22d814658852b0237922bbfb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f3f5a7d5afd41ceb534f34d36810852"
I20260619 16:43:24.054335 2301 raft_consensus.cc:3037] T 15ab05270823492481cad281abdbb2f9 P 0f3f5a7d5afd41ceb534f34d36810852 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:43:24.054318 2445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15ab05270823492481cad281abdbb2f9" candidate_uuid: "8cb0bbb22d814658852b0237922bbfb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c9be31194f04e75874e50b6af75e143"
I20260619 16:43:24.054589 2445 raft_consensus.cc:3037] T 15ab05270823492481cad281abdbb2f9 P 4c9be31194f04e75874e50b6af75e143 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:43:24.056366 2301 raft_consensus.cc:2454] T 15ab05270823492481cad281abdbb2f9 P 0f3f5a7d5afd41ceb534f34d36810852 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8cb0bbb22d814658852b0237922bbfb3 in term 1.
I20260619 16:43:24.056756 2510 leader_election.cc:302] T 15ab05270823492481cad281abdbb2f9 P 8cb0bbb22d814658852b0237922bbfb3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f3f5a7d5afd41ceb534f34d36810852, 8cb0bbb22d814658852b0237922bbfb3; no voters:
I20260619 16:43:24.056877 2445 raft_consensus.cc:2454] T 15ab05270823492481cad281abdbb2f9 P 4c9be31194f04e75874e50b6af75e143 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8cb0bbb22d814658852b0237922bbfb3 in term 1.
I20260619 16:43:24.057030 2715 raft_consensus.cc:2781] T 15ab05270823492481cad281abdbb2f9 P 8cb0bbb22d814658852b0237922bbfb3 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:43:24.057488 2715 raft_consensus.cc:686] T 15ab05270823492481cad281abdbb2f9 P 8cb0bbb22d814658852b0237922bbfb3 [term 1 LEADER]: Becoming Leader. State: Replica: 8cb0bbb22d814658852b0237922bbfb3, State: Running, Role: LEADER
I20260619 16:43:24.057817 2715 consensus_queue.cc:237] T 15ab05270823492481cad281abdbb2f9 P 8cb0bbb22d814658852b0237922bbfb3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f3f5a7d5afd41ceb534f34d36810852" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35421 } } peers { permanent_uuid: "4c9be31194f04e75874e50b6af75e143" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41989 } } peers { permanent_uuid: "8cb0bbb22d814658852b0237922bbfb3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45671 } }
I20260619 16:43:24.060395 2144 catalog_manager.cc:5495] T 15ab05270823492481cad281abdbb2f9 P 8cb0bbb22d814658852b0237922bbfb3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8cb0bbb22d814658852b0237922bbfb3 (127.0.97.3). New cstate: current_term: 1 leader_uuid: "8cb0bbb22d814658852b0237922bbfb3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f3f5a7d5afd41ceb534f34d36810852" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35421 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4c9be31194f04e75874e50b6af75e143" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41989 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8cb0bbb22d814658852b0237922bbfb3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45671 } health_report { overall_health: HEALTHY } } }
I20260619 16:43:24.109681 2741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:24.104759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55982 (local address 127.0.97.3:45671)
0619 16:43:24.105230 (+ 471us) server_negotiation.cc:206] Beginning negotiation
0619 16:43:24.105234 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:43:24.105251 (+ 17us) server_negotiation.cc:407] Connection header received
0619 16:43:24.105303 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:43:24.105307 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:43:24.105376 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:43:24.105450 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 16:43:24.105725 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:24.106660 (+ 935us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:24.107950 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:24.108463 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:24.108574 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:43:24.108987 (+ 413us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 16:43:24.109094 (+ 107us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 16:43:24.109172 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0619 16:43:24.109308 (+ 136us) server_negotiation.cc:299] Negotiation successful
0619 16:43:24.109385 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":380,"thread_start_us":267,"threads_started":1}
I20260619 16:43:24.119292 2445 raft_consensus.cc:1261] T 15ab05270823492481cad281abdbb2f9 P 4c9be31194f04e75874e50b6af75e143 [term 1 FOLLOWER]: Refusing update from remote peer 8cb0bbb22d814658852b0237922bbfb3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:43:24.119292 2301 raft_consensus.cc:1261] T 15ab05270823492481cad281abdbb2f9 P 0f3f5a7d5afd41ceb534f34d36810852 [term 1 FOLLOWER]: Refusing update from remote peer 8cb0bbb22d814658852b0237922bbfb3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:43:24.119825 2681 consensus_queue.cc:1035] T 15ab05270823492481cad281abdbb2f9 P 8cb0bbb22d814658852b0237922bbfb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c9be31194f04e75874e50b6af75e143" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:43:24.120080 2715 consensus_queue.cc:1035] T 15ab05270823492481cad281abdbb2f9 P 8cb0bbb22d814658852b0237922bbfb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f3f5a7d5afd41ceb534f34d36810852" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:43:24.177151 2145 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49432:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:43:24.177347 2145 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49432:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:43:24.180119 2145 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 58e545bbf9084b46bd00d2e25bf47bb3: Sending DeleteTablet for 3 replicas of tablet 15ab05270823492481cad281abdbb2f9
I20260619 16:43:24.180857 2280 tablet_service.cc:1514] Processing DeleteTablet for tablet 15ab05270823492481cad281abdbb2f9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:43:24 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:60222
I20260619 16:43:24.181010 2420 tablet_service.cc:1514] Processing DeleteTablet for tablet 15ab05270823492481cad281abdbb2f9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:43:24 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:33140
I20260619 16:43:24.181137 2721 tablet_replica.cc:330] T 15ab05270823492481cad281abdbb2f9 P 0f3f5a7d5afd41ceb534f34d36810852: stopping tablet replica
I20260619 16:43:24.181288 2720 tablet_replica.cc:330] T 15ab05270823492481cad281abdbb2f9 P 4c9be31194f04e75874e50b6af75e143: stopping tablet replica
I20260619 16:43:24.181331 2721 raft_consensus.cc:2229] T 15ab05270823492481cad281abdbb2f9 P 0f3f5a7d5afd41ceb534f34d36810852 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 16:43:24.181475 2720 raft_consensus.cc:2229] T 15ab05270823492481cad281abdbb2f9 P 4c9be31194f04e75874e50b6af75e143 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 16:43:24.181548 2721 raft_consensus.cc:2258] T 15ab05270823492481cad281abdbb2f9 P 0f3f5a7d5afd41ceb534f34d36810852 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:43:24.181586 2558 tablet_service.cc:1514] Processing DeleteTablet for tablet 15ab05270823492481cad281abdbb2f9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:43:24 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:32966
I20260619 16:43:24.181638 2720 raft_consensus.cc:2258] T 15ab05270823492481cad281abdbb2f9 P 4c9be31194f04e75874e50b6af75e143 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:43:24.181861 2722 tablet_replica.cc:330] T 15ab05270823492481cad281abdbb2f9 P 8cb0bbb22d814658852b0237922bbfb3: stopping tablet replica
I20260619 16:43:24.182045 2722 raft_consensus.cc:2229] T 15ab05270823492481cad281abdbb2f9 P 8cb0bbb22d814658852b0237922bbfb3 [term 1 LEADER]: Raft consensus shutting down.
I20260619 16:43:24.181875 2145 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49432:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260619 16:43:24.182348 2722 raft_consensus.cc:2258] T 15ab05270823492481cad281abdbb2f9 P 8cb0bbb22d814658852b0237922bbfb3 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260619 16:43:24.182483 2145 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 16:43:24.182634 2721 ts_tablet_manager.cc:1905] T 15ab05270823492481cad281abdbb2f9 P 0f3f5a7d5afd41ceb534f34d36810852: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:43:24.182811 2720 ts_tablet_manager.cc:1905] T 15ab05270823492481cad281abdbb2f9 P 4c9be31194f04e75874e50b6af75e143: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:43:24.183559 2722 ts_tablet_manager.cc:1905] T 15ab05270823492481cad281abdbb2f9 P 8cb0bbb22d814658852b0237922bbfb3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:43:24.189149 2280 tablet_service.cc:1467] Processing CreateTablet for tablet 3e193904a8014babb7a944bc18cb7a6e (DEFAULT_TABLE table=test-table [id=5dad1dad468a4579a9e341ff6bba230d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:43:24.189247 2721 ts_tablet_manager.cc:1918] T 15ab05270823492481cad281abdbb2f9 P 0f3f5a7d5afd41ceb534f34d36810852: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 16:43:24.189394 2721 log.cc:1196] T 15ab05270823492481cad281abdbb2f9 P 0f3f5a7d5afd41ceb534f34d36810852: Deleting WAL directory at /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887385903558-388-0/minicluster-data/ts-0/wal/wals/15ab05270823492481cad281abdbb2f9
I20260619 16:43:24.189321 2420 tablet_service.cc:1467] Processing CreateTablet for tablet 3e193904a8014babb7a944bc18cb7a6e (DEFAULT_TABLE table=test-table [id=5dad1dad468a4579a9e341ff6bba230d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:43:24.189571 2280 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3e193904a8014babb7a944bc18cb7a6e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:43:24.189689 2420 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3e193904a8014babb7a944bc18cb7a6e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:43:24.189802 2721 ts_tablet_manager.cc:1939] T 15ab05270823492481cad281abdbb2f9 P 0f3f5a7d5afd41ceb534f34d36810852: Deleting consensus metadata
I20260619 16:43:24.191753 2720 ts_tablet_manager.cc:1918] T 15ab05270823492481cad281abdbb2f9 P 4c9be31194f04e75874e50b6af75e143: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 16:43:24.191886 2720 log.cc:1196] T 15ab05270823492481cad281abdbb2f9 P 4c9be31194f04e75874e50b6af75e143: Deleting WAL directory at /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887385903558-388-0/minicluster-data/ts-1/wal/wals/15ab05270823492481cad281abdbb2f9
I20260619 16:43:24.192224 2720 ts_tablet_manager.cc:1939] T 15ab05270823492481cad281abdbb2f9 P 4c9be31194f04e75874e50b6af75e143: Deleting consensus metadata
I20260619 16:43:24.193205 2130 catalog_manager.cc:4841] TS 4c9be31194f04e75874e50b6af75e143 (127.0.97.2:41989): tablet 15ab05270823492481cad281abdbb2f9 (table test-table [id=5415dc5964564d4da2869209da4705bd]) successfully deleted
I20260619 16:43:24.193463 2129 catalog_manager.cc:4841] TS 0f3f5a7d5afd41ceb534f34d36810852 (127.0.97.1:35421): tablet 15ab05270823492481cad281abdbb2f9 (table test-table [id=5415dc5964564d4da2869209da4705bd]) successfully deleted
I20260619 16:43:24.193820 2732 tablet_bootstrap.cc:492] T 3e193904a8014babb7a944bc18cb7a6e P 0f3f5a7d5afd41ceb534f34d36810852: Bootstrap starting.
I20260619 16:43:24.194550 2558 tablet_service.cc:1467] Processing CreateTablet for tablet 3e193904a8014babb7a944bc18cb7a6e (DEFAULT_TABLE table=test-table [id=5dad1dad468a4579a9e341ff6bba230d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:43:24.194895 2558 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3e193904a8014babb7a944bc18cb7a6e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:43:24.194980 2722 ts_tablet_manager.cc:1918] T 15ab05270823492481cad281abdbb2f9 P 8cb0bbb22d814658852b0237922bbfb3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 16:43:24.195119 2722 log.cc:1196] T 15ab05270823492481cad281abdbb2f9 P 8cb0bbb22d814658852b0237922bbfb3: Deleting WAL directory at /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887385903558-388-0/minicluster-data/ts-2/wal/wals/15ab05270823492481cad281abdbb2f9
I20260619 16:43:24.195330 2732 tablet_bootstrap.cc:654] T 3e193904a8014babb7a944bc18cb7a6e P 0f3f5a7d5afd41ceb534f34d36810852: Neither blocks nor log segments found. Creating new log.
I20260619 16:43:24.195534 2722 ts_tablet_manager.cc:1939] T 15ab05270823492481cad281abdbb2f9 P 8cb0bbb22d814658852b0237922bbfb3: Deleting consensus metadata
I20260619 16:43:24.196909 2129 catalog_manager.cc:4841] TS 8cb0bbb22d814658852b0237922bbfb3 (127.0.97.3:45671): tablet 15ab05270823492481cad281abdbb2f9 (table test-table [id=5415dc5964564d4da2869209da4705bd]) successfully deleted
I20260619 16:43:24.198052 2732 tablet_bootstrap.cc:492] T 3e193904a8014babb7a944bc18cb7a6e P 0f3f5a7d5afd41ceb534f34d36810852: No bootstrap required, opened a new log
I20260619 16:43:24.198071 2733 tablet_bootstrap.cc:492] T 3e193904a8014babb7a944bc18cb7a6e P 4c9be31194f04e75874e50b6af75e143: Bootstrap starting.
I20260619 16:43:24.198269 2732 ts_tablet_manager.cc:1397] T 3e193904a8014babb7a944bc18cb7a6e P 0f3f5a7d5afd41ceb534f34d36810852: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.003s
I20260619 16:43:24.199034 2732 raft_consensus.cc:348] T 3e193904a8014babb7a944bc18cb7a6e P 0f3f5a7d5afd41ceb534f34d36810852 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f3f5a7d5afd41ceb534f34d36810852" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35421 } } peers { permanent_uuid: "4c9be31194f04e75874e50b6af75e143" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41989 } } peers { permanent_uuid: "8cb0bbb22d814658852b0237922bbfb3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45671 } }
I20260619 16:43:24.199334 2732 raft_consensus.cc:374] T 3e193904a8014babb7a944bc18cb7a6e P 0f3f5a7d5afd41ceb534f34d36810852 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:43:24.199465 2732 raft_consensus.cc:729] T 3e193904a8014babb7a944bc18cb7a6e P 0f3f5a7d5afd41ceb534f34d36810852 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f3f5a7d5afd41ceb534f34d36810852, State: Initialized, Role: FOLLOWER
I20260619 16:43:24.199654 2733 tablet_bootstrap.cc:654] T 3e193904a8014babb7a944bc18cb7a6e P 4c9be31194f04e75874e50b6af75e143: Neither blocks nor log segments found. Creating new log.
I20260619 16:43:24.199685 2732 consensus_queue.cc:260] T 3e193904a8014babb7a944bc18cb7a6e P 0f3f5a7d5afd41ceb534f34d36810852 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f3f5a7d5afd41ceb534f34d36810852" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35421 } } peers { permanent_uuid: "4c9be31194f04e75874e50b6af75e143" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41989 } } peers { permanent_uuid: "8cb0bbb22d814658852b0237922bbfb3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45671 } }
I20260619 16:43:24.200171 2732 ts_tablet_manager.cc:1428] T 3e193904a8014babb7a944bc18cb7a6e P 0f3f5a7d5afd41ceb534f34d36810852: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20260619 16:43:24.201148 2731 tablet_bootstrap.cc:492] T 3e193904a8014babb7a944bc18cb7a6e P 8cb0bbb22d814658852b0237922bbfb3: Bootstrap starting.
I20260619 16:43:24.201411 2733 tablet_bootstrap.cc:492] T 3e193904a8014babb7a944bc18cb7a6e P 4c9be31194f04e75874e50b6af75e143: No bootstrap required, opened a new log
I20260619 16:43:24.201558 2733 ts_tablet_manager.cc:1397] T 3e193904a8014babb7a944bc18cb7a6e P 4c9be31194f04e75874e50b6af75e143: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260619 16:43:24.202283 2733 raft_consensus.cc:348] T 3e193904a8014babb7a944bc18cb7a6e P 4c9be31194f04e75874e50b6af75e143 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f3f5a7d5afd41ceb534f34d36810852" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35421 } } peers { permanent_uuid: "4c9be31194f04e75874e50b6af75e143" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41989 } } peers { permanent_uuid: "8cb0bbb22d814658852b0237922bbfb3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45671 } }
I20260619 16:43:24.202518 2733 raft_consensus.cc:374] T 3e193904a8014babb7a944bc18cb7a6e P 4c9be31194f04e75874e50b6af75e143 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:43:24.202587 2733 raft_consensus.cc:729] T 3e193904a8014babb7a944bc18cb7a6e P 4c9be31194f04e75874e50b6af75e143 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c9be31194f04e75874e50b6af75e143, State: Initialized, Role: FOLLOWER
I20260619 16:43:24.202642 2731 tablet_bootstrap.cc:654] T 3e193904a8014babb7a944bc18cb7a6e P 8cb0bbb22d814658852b0237922bbfb3: Neither blocks nor log segments found. Creating new log.
I20260619 16:43:24.202839 2733 consensus_queue.cc:260] T 3e193904a8014babb7a944bc18cb7a6e P 4c9be31194f04e75874e50b6af75e143 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f3f5a7d5afd41ceb534f34d36810852" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35421 } } peers { permanent_uuid: "4c9be31194f04e75874e50b6af75e143" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41989 } } peers { permanent_uuid: "8cb0bbb22d814658852b0237922bbfb3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45671 } }
I20260619 16:43:24.203327 2733 ts_tablet_manager.cc:1428] T 3e193904a8014babb7a944bc18cb7a6e P 4c9be31194f04e75874e50b6af75e143: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260619 16:43:24.204861 2731 tablet_bootstrap.cc:492] T 3e193904a8014babb7a944bc18cb7a6e P 8cb0bbb22d814658852b0237922bbfb3: No bootstrap required, opened a new log
I20260619 16:43:24.205008 2731 ts_tablet_manager.cc:1397] T 3e193904a8014babb7a944bc18cb7a6e P 8cb0bbb22d814658852b0237922bbfb3: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260619 16:43:24.205622 2731 raft_consensus.cc:348] T 3e193904a8014babb7a944bc18cb7a6e P 8cb0bbb22d814658852b0237922bbfb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f3f5a7d5afd41ceb534f34d36810852" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35421 } } peers { permanent_uuid: "4c9be31194f04e75874e50b6af75e143" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41989 } } peers { permanent_uuid: "8cb0bbb22d814658852b0237922bbfb3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45671 } }
I20260619 16:43:24.205878 2731 raft_consensus.cc:374] T 3e193904a8014babb7a944bc18cb7a6e P 8cb0bbb22d814658852b0237922bbfb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:43:24.205960 2731 raft_consensus.cc:729] T 3e193904a8014babb7a944bc18cb7a6e P 8cb0bbb22d814658852b0237922bbfb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8cb0bbb22d814658852b0237922bbfb3, State: Initialized, Role: FOLLOWER
I20260619 16:43:24.206204 2731 consensus_queue.cc:260] T 3e193904a8014babb7a944bc18cb7a6e P 8cb0bbb22d814658852b0237922bbfb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f3f5a7d5afd41ceb534f34d36810852" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35421 } } peers { permanent_uuid: "4c9be31194f04e75874e50b6af75e143" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41989 } } peers { permanent_uuid: "8cb0bbb22d814658852b0237922bbfb3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45671 } }
I20260619 16:43:24.206670 2731 ts_tablet_manager.cc:1428] T 3e193904a8014babb7a944bc18cb7a6e P 8cb0bbb22d814658852b0237922bbfb3: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260619 16:43:24.255721 2681 raft_consensus.cc:482] T 3e193904a8014babb7a944bc18cb7a6e P 8cb0bbb22d814658852b0237922bbfb3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 16:43:24.255923 2681 raft_consensus.cc:504] T 3e193904a8014babb7a944bc18cb7a6e P 8cb0bbb22d814658852b0237922bbfb3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f3f5a7d5afd41ceb534f34d36810852" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35421 } } peers { permanent_uuid: "4c9be31194f04e75874e50b6af75e143" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41989 } } peers { permanent_uuid: "8cb0bbb22d814658852b0237922bbfb3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45671 } }
I20260619 16:43:24.256594 2681 leader_election.cc:288] T 3e193904a8014babb7a944bc18cb7a6e P 8cb0bbb22d814658852b0237922bbfb3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0f3f5a7d5afd41ceb534f34d36810852 (127.0.97.1:35421), 4c9be31194f04e75874e50b6af75e143 (127.0.97.2:41989)
I20260619 16:43:24.257102 2300 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e193904a8014babb7a944bc18cb7a6e" candidate_uuid: "8cb0bbb22d814658852b0237922bbfb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f3f5a7d5afd41ceb534f34d36810852" is_pre_election: true
I20260619 16:43:24.257131 2444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e193904a8014babb7a944bc18cb7a6e" candidate_uuid: "8cb0bbb22d814658852b0237922bbfb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c9be31194f04e75874e50b6af75e143" is_pre_election: true
I20260619 16:43:24.257346 2444 raft_consensus.cc:2454] T 3e193904a8014babb7a944bc18cb7a6e P 4c9be31194f04e75874e50b6af75e143 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8cb0bbb22d814658852b0237922bbfb3 in term 0.
I20260619 16:43:24.257350 2300 raft_consensus.cc:2454] T 3e193904a8014babb7a944bc18cb7a6e P 0f3f5a7d5afd41ceb534f34d36810852 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8cb0bbb22d814658852b0237922bbfb3 in term 0.
I20260619 16:43:24.257723 2511 leader_election.cc:302] T 3e193904a8014babb7a944bc18cb7a6e P 8cb0bbb22d814658852b0237922bbfb3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c9be31194f04e75874e50b6af75e143, 8cb0bbb22d814658852b0237922bbfb3; no voters:
I20260619 16:43:24.258021 2681 raft_consensus.cc:2781] T 3e193904a8014babb7a944bc18cb7a6e P 8cb0bbb22d814658852b0237922bbfb3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 16:43:24.258133 2681 raft_consensus.cc:482] T 3e193904a8014babb7a944bc18cb7a6e P 8cb0bbb22d814658852b0237922bbfb3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 16:43:24.258219 2681 raft_consensus.cc:3037] T 3e193904a8014babb7a944bc18cb7a6e P 8cb0bbb22d814658852b0237922bbfb3 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:43:24.260232 2681 raft_consensus.cc:504] T 3e193904a8014babb7a944bc18cb7a6e P 8cb0bbb22d814658852b0237922bbfb3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f3f5a7d5afd41ceb534f34d36810852" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35421 } } peers { permanent_uuid: "4c9be31194f04e75874e50b6af75e143" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41989 } } peers { permanent_uuid: "8cb0bbb22d814658852b0237922bbfb3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45671 } }
I20260619 16:43:24.260731 2681 leader_election.cc:288] T 3e193904a8014babb7a944bc18cb7a6e P 8cb0bbb22d814658852b0237922bbfb3 [CANDIDATE]: Term 1 election: Requested vote from peers 0f3f5a7d5afd41ceb534f34d36810852 (127.0.97.1:35421), 4c9be31194f04e75874e50b6af75e143 (127.0.97.2:41989)
I20260619 16:43:24.261212 2300 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e193904a8014babb7a944bc18cb7a6e" candidate_uuid: "8cb0bbb22d814658852b0237922bbfb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f3f5a7d5afd41ceb534f34d36810852"
I20260619 16:43:24.261212 2444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e193904a8014babb7a944bc18cb7a6e" candidate_uuid: "8cb0bbb22d814658852b0237922bbfb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c9be31194f04e75874e50b6af75e143"
I20260619 16:43:24.261430 2300 raft_consensus.cc:3037] T 3e193904a8014babb7a944bc18cb7a6e P 0f3f5a7d5afd41ceb534f34d36810852 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:43:24.261439 2444 raft_consensus.cc:3037] T 3e193904a8014babb7a944bc18cb7a6e P 4c9be31194f04e75874e50b6af75e143 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:43:24.264181 2300 raft_consensus.cc:2454] T 3e193904a8014babb7a944bc18cb7a6e P 0f3f5a7d5afd41ceb534f34d36810852 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8cb0bbb22d814658852b0237922bbfb3 in term 1.
I20260619 16:43:24.264181 2444 raft_consensus.cc:2454] T 3e193904a8014babb7a944bc18cb7a6e P 4c9be31194f04e75874e50b6af75e143 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8cb0bbb22d814658852b0237922bbfb3 in term 1.
I20260619 16:43:24.264621 2510 leader_election.cc:302] T 3e193904a8014babb7a944bc18cb7a6e P 8cb0bbb22d814658852b0237922bbfb3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 0f3f5a7d5afd41ceb534f34d36810852, 4c9be31194f04e75874e50b6af75e143, 8cb0bbb22d814658852b0237922bbfb3; no voters:
I20260619 16:43:24.264899 2681 raft_consensus.cc:2781] T 3e193904a8014babb7a944bc18cb7a6e P 8cb0bbb22d814658852b0237922bbfb3 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:43:24.265026 2681 raft_consensus.cc:686] T 3e193904a8014babb7a944bc18cb7a6e P 8cb0bbb22d814658852b0237922bbfb3 [term 1 LEADER]: Becoming Leader. State: Replica: 8cb0bbb22d814658852b0237922bbfb3, State: Running, Role: LEADER
I20260619 16:43:24.265265 2681 consensus_queue.cc:237] T 3e193904a8014babb7a944bc18cb7a6e P 8cb0bbb22d814658852b0237922bbfb3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f3f5a7d5afd41ceb534f34d36810852" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35421 } } peers { permanent_uuid: "4c9be31194f04e75874e50b6af75e143" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41989 } } peers { permanent_uuid: "8cb0bbb22d814658852b0237922bbfb3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45671 } }
I20260619 16:43:24.267714 2144 catalog_manager.cc:5495] T 3e193904a8014babb7a944bc18cb7a6e P 8cb0bbb22d814658852b0237922bbfb3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8cb0bbb22d814658852b0237922bbfb3 (127.0.97.3). New cstate: current_term: 1 leader_uuid: "8cb0bbb22d814658852b0237922bbfb3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f3f5a7d5afd41ceb534f34d36810852" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35421 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4c9be31194f04e75874e50b6af75e143" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41989 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8cb0bbb22d814658852b0237922bbfb3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45671 } health_report { overall_health: HEALTHY } } }
I20260619 16:43:24.293349 2444 raft_consensus.cc:1261] T 3e193904a8014babb7a944bc18cb7a6e P 4c9be31194f04e75874e50b6af75e143 [term 1 FOLLOWER]: Refusing update from remote peer 8cb0bbb22d814658852b0237922bbfb3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:43:24.293356 2300 raft_consensus.cc:1261] T 3e193904a8014babb7a944bc18cb7a6e P 0f3f5a7d5afd41ceb534f34d36810852 [term 1 FOLLOWER]: Refusing update from remote peer 8cb0bbb22d814658852b0237922bbfb3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:43:24.293879 2708 consensus_queue.cc:1035] T 3e193904a8014babb7a944bc18cb7a6e P 8cb0bbb22d814658852b0237922bbfb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f3f5a7d5afd41ceb534f34d36810852" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:43:24.294153 2715 consensus_queue.cc:1035] T 3e193904a8014babb7a944bc18cb7a6e P 8cb0bbb22d814658852b0237922bbfb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c9be31194f04e75874e50b6af75e143" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260619 16:43:24.311589 2700 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260619 16:43:25.886650 2144 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49432:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:43:25.886880 2144 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49432:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:43:25.892452 2144 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 58e545bbf9084b46bd00d2e25bf47bb3: Sending DeleteTablet for 3 replicas of tablet 3e193904a8014babb7a944bc18cb7a6e
I20260619 16:43:25.893451 2558 tablet_service.cc:1514] Processing DeleteTablet for tablet 3e193904a8014babb7a944bc18cb7a6e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:43:25 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:32966
I20260619 16:43:25.893527 2280 tablet_service.cc:1514] Processing DeleteTablet for tablet 3e193904a8014babb7a944bc18cb7a6e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:43:25 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:60222
I20260619 16:43:25.894286 2770 tablet_replica.cc:330] T 3e193904a8014babb7a944bc18cb7a6e P 8cb0bbb22d814658852b0237922bbfb3: stopping tablet replica
I20260619 16:43:25.894261 2420 tablet_service.cc:1514] Processing DeleteTablet for tablet 3e193904a8014babb7a944bc18cb7a6e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:43:25 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:33140
I20260619 16:43:25.894480 2770 raft_consensus.cc:2229] T 3e193904a8014babb7a944bc18cb7a6e P 8cb0bbb22d814658852b0237922bbfb3 [term 1 LEADER]: Raft consensus shutting down.
I20260619 16:43:25.894621 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu pid 2216
I20260619 16:43:25.894855 2770 raft_consensus.cc:2258] T 3e193904a8014babb7a944bc18cb7a6e P 8cb0bbb22d814658852b0237922bbfb3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:43:25.896369 2770 ts_tablet_manager.cc:1905] T 3e193904a8014babb7a944bc18cb7a6e P 8cb0bbb22d814658852b0237922bbfb3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:43:25.902093 2771 tablet_replica.cc:330] T 3e193904a8014babb7a944bc18cb7a6e P 0f3f5a7d5afd41ceb534f34d36810852: stopping tablet replica
I20260619 16:43:25.902302 2771 raft_consensus.cc:2229] T 3e193904a8014babb7a944bc18cb7a6e P 0f3f5a7d5afd41ceb534f34d36810852 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 16:43:25.902530 2771 raft_consensus.cc:2258] T 3e193904a8014babb7a944bc18cb7a6e P 0f3f5a7d5afd41ceb534f34d36810852 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:43:25.903322 2771 ts_tablet_manager.cc:1905] T 3e193904a8014babb7a944bc18cb7a6e P 0f3f5a7d5afd41ceb534f34d36810852: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:43:25.906140 2772 tablet_replica.cc:330] T 3e193904a8014babb7a944bc18cb7a6e P 4c9be31194f04e75874e50b6af75e143: stopping tablet replica
I20260619 16:43:25.906311 2772 raft_consensus.cc:2229] T 3e193904a8014babb7a944bc18cb7a6e P 4c9be31194f04e75874e50b6af75e143 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 16:43:25.906488 2772 raft_consensus.cc:2258] T 3e193904a8014babb7a944bc18cb7a6e P 4c9be31194f04e75874e50b6af75e143 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:43:25.909138 2772 ts_tablet_manager.cc:1905] T 3e193904a8014babb7a944bc18cb7a6e P 4c9be31194f04e75874e50b6af75e143: Deleting tablet data with delete state TABLET_DATA_DELETED
Jun 19 16:43:25 dist-test-slave-wvz7 krb5kdc[2100](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887403, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260619 16:43:25.917251 2774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:25.896550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56030 (local address 127.0.97.1:35421)
0619 16:43:25.897057 (+ 507us) server_negotiation.cc:206] Beginning negotiation
0619 16:43:25.897063 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:43:25.897086 (+ 23us) server_negotiation.cc:407] Connection header received
0619 16:43:25.897144 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:43:25.897150 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:43:25.897228 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:43:25.899116 (+ 1888us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:43:25.899290 (+ 174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:25.900643 (+ 1353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:25.902622 (+ 1979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:25.903383 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:25.904146 (+ 763us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:43:25.910468 (+ 6322us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:43:25.910484 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:43:25.910500 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:43:25.910534 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:43:25.913453 (+ 2919us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:43:25.914284 (+ 831us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:43:25.914294 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:43:25.914298 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:43:25.914479 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:43:25.914938 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:43:25.914943 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:43:25.914947 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:43:25.915426 (+ 479us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:43:25.915737 (+ 311us) server_negotiation.cc:1035] Waiting for connection context
0619 16:43:25.916802 (+ 1065us) server_negotiation.cc:299] Negotiation successful
0619 16:43:25.916974 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":367,"thread_start_us":234,"threads_started":1}
W20260619 16:43:25.918192 2346 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56030
W20260619 16:43:25.918835 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu pid 2216: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 16:43:25.919025 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu with pid 2216
I20260619 16:43:25.994882 2770 ts_tablet_manager.cc:1918] T 3e193904a8014babb7a944bc18cb7a6e P 8cb0bbb22d814658852b0237922bbfb3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260619 16:43:25.994884 2772 ts_tablet_manager.cc:1918] T 3e193904a8014babb7a944bc18cb7a6e P 4c9be31194f04e75874e50b6af75e143: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260619 16:43:25.995065 2772 log.cc:1196] T 3e193904a8014babb7a944bc18cb7a6e P 4c9be31194f04e75874e50b6af75e143: Deleting WAL directory at /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887385903558-388-0/minicluster-data/ts-1/wal/wals/3e193904a8014babb7a944bc18cb7a6e
I20260619 16:43:25.995080 2770 log.cc:1196] T 3e193904a8014babb7a944bc18cb7a6e P 8cb0bbb22d814658852b0237922bbfb3: Deleting WAL directory at /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887385903558-388-0/minicluster-data/ts-2/wal/wals/3e193904a8014babb7a944bc18cb7a6e
I20260619 16:43:25.995507 2770 ts_tablet_manager.cc:1939] T 3e193904a8014babb7a944bc18cb7a6e P 8cb0bbb22d814658852b0237922bbfb3: Deleting consensus metadata
I20260619 16:43:25.995529 2772 ts_tablet_manager.cc:1939] T 3e193904a8014babb7a944bc18cb7a6e P 4c9be31194f04e75874e50b6af75e143: Deleting consensus metadata
I20260619 16:43:25.996626 2130 catalog_manager.cc:4841] TS 4c9be31194f04e75874e50b6af75e143 (127.0.97.2:41989): tablet 3e193904a8014babb7a944bc18cb7a6e (table test-table [id=5dad1dad468a4579a9e341ff6bba230d]) successfully deleted
I20260619 16:43:25.996692 2129 catalog_manager.cc:4841] TS 8cb0bbb22d814658852b0237922bbfb3 (127.0.97.3:45671): tablet 3e193904a8014babb7a944bc18cb7a6e (table test-table [id=5dad1dad468a4579a9e341ff6bba230d]) successfully deleted
W20260619 16:43:26.007406 2129 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.97.1:35421 (error 108)
I20260619 16:43:26.007639 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu pid 2355
I20260619 16:43:26.009724 2777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:26.008990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:35421 (local address 127.0.0.1:56044)
0619 16:43:26.009442 (+ 452us) negotiation.cc:107] Waiting for socket to connect
0619 16:43:26.009543 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:35421: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":336,"thread_start_us":219,"threads_started":1}
W20260619 16:43:26.009932 2129 catalog_manager.cc:4584] TS 0f3f5a7d5afd41ceb534f34d36810852 (127.0.97.1:35421): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 3e193904a8014babb7a944bc18cb7a6e: Network error: Client connection negotiation failed: client connection to 127.0.97.1:35421: connect: Connection refused (error 111)
Jun 19 16:43:26 dist-test-slave-wvz7 krb5kdc[2100](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887403, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260619 16:43:26.022601 2776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:26.008514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58984 (local address 127.0.97.2:41989)
0619 16:43:26.008966 (+ 452us) server_negotiation.cc:206] Beginning negotiation
0619 16:43:26.008970 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:43:26.008986 (+ 16us) server_negotiation.cc:407] Connection header received
0619 16:43:26.009034 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:43:26.009038 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:43:26.009098 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:43:26.009198 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:43:26.010484 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:26.011220 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:26.012429 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:26.012824 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:26.012928 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:43:26.016853 (+ 3925us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:43:26.016867 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:43:26.016882 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:43:26.016915 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:43:26.019779 (+ 2864us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:43:26.020498 (+ 719us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:43:26.020505 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:43:26.020508 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:43:26.020654 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:43:26.021009 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:43:26.021015 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:43:26.021018 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:43:26.021390 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:43:26.021670 (+ 280us) server_negotiation.cc:1035] Waiting for connection context
0619 16:43:26.022002 (+ 332us) server_negotiation.cc:299] Negotiation successful
0619 16:43:26.022238 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":341,"thread_start_us":232,"threads_started":1}
W20260619 16:43:26.023331 2485 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58984
W20260619 16:43:26.024003 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu pid 2355: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 16:43:26.024155 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu with pid 2355
I20260619 16:43:26.037880 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu pid 2494
Jun 19 16:43:26 dist-test-slave-wvz7 krb5kdc[2100](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887403, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260619 16:43:26.053293 2778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:26.038637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55998 (local address 127.0.97.3:45671)
0619 16:43:26.039140 (+ 503us) server_negotiation.cc:206] Beginning negotiation
0619 16:43:26.039145 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:43:26.039161 (+ 16us) server_negotiation.cc:407] Connection header received
0619 16:43:26.039210 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:43:26.039214 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:43:26.039311 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:43:26.039401 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:43:26.040789 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:26.041599 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:26.042953 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:26.043410 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:26.043517 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:43:26.047799 (+ 4282us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:43:26.047814 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:43:26.047830 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:43:26.047867 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:43:26.050750 (+ 2883us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:43:26.051383 (+ 633us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:43:26.051391 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:43:26.051394 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:43:26.051513 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:43:26.051868 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:43:26.051872 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:43:26.051874 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:43:26.052220 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:43:26.052500 (+ 280us) server_negotiation.cc:1035] Waiting for connection context
0619 16:43:26.052753 (+ 253us) server_negotiation.cc:299] Negotiation successful
0619 16:43:26.052960 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":392,"thread_start_us":266,"threads_started":1}
W20260619 16:43:26.053989 2624 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55998
W20260619 16:43:26.054646 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu pid 2494: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 16:43:26.054802 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu with pid 2494
I20260619 16:43:26.063220 2777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:26.062806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:35421 (local address 127.0.0.1:56050)
0619 16:43:26.062994 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0619 16:43:26.063084 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:35421: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260619 16:43:26.064136 2777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:26.063802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:35421 (local address 127.0.0.1:56066)
0619 16:43:26.063935 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0619 16:43:26.064011 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:35421: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260619 16:43:26.074259 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu pid 2113
I20260619 16:43:26.075383 2185 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:43:26.186759 2777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:26.186257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:35421 (local address 127.0.0.1:56068)
0619 16:43:26.186483 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0619 16:43:26.186590 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:35421: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260619 16:43:26.186928 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu with pid 2113
2026-06-19T16:43:26Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (8195 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887385903558-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 16:43:26 dist-test-slave-wvz7 krb5kdc[2782](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 16:43:26 dist-test-slave-wvz7 krb5kdc[2782](info): set up 2 sockets
Jun 19 16:43:26 dist-test-slave-wvz7 krb5kdc[2782](info): commencing operation
krb5kdc: starting...
W20260619 16:43:27.181947 388 mini_kdc.cc:121] Time spent starting KDC: real 0.956s user 0.002s sys 0.013s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887385903558-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887385903558-388-0/krb5kdc/test-user.keytab.
Jun 19 16:43:27 dist-test-slave-wvz7 krb5kdc[2782](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887407, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T16:43:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T16:43:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887385903558-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887385903558-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260619 16:43:27.351081 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu
/tmp/dist-test-taskEs2EH8/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887385903558-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887385903558-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887385903558-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887385903558-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45577
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887385903558-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36959
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:45577
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887385903558-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887385903558-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887385903558-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:43:27.652669 2795 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:43:27.652949 2795 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:43:27.653070 2795 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:43:27.660241 2795 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:43:27.660344 2795 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:43:27.660398 2795 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:43:27.660473 2795 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:43:27.660531 2795 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:43:27.670203 2795 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36959
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887385903558-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887385903558-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:45577
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887385903558-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45577
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887385903558-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887385903558-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:43:27.671478 2795 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:43:27.672739 2795 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:43:27.683808 2801 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:43:27.683846 2803 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:43:27.683872 2800 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:43:27.685220 2795 server_base.cc:1034] running on GCE node
I20260619 16:43:27.686108 2795 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:43:27.687525 2795 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:43:27.688752 2795 hybrid_clock.cc:648] HybridClock initialized: now 1781887407688687 us; error 46 us; skew 500 ppm
I20260619 16:43:27.689085 2795 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:43:27 dist-test-slave-wvz7 krb5kdc[2782](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887407, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:43:27.692060 2795 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260619 16:43:27.693358 2795 webserver.cc:460] Webserver started at http://127.0.97.62:34205/ using document root <none> and password file <none>
I20260619 16:43:27.694034 2795 fs_manager.cc:359] Metadata directory not provided
I20260619 16:43:27.694145 2795 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:43:27.694415 2795 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:43:27.697228 2795 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887385903558-388-0/minicluster-data/master-0/data/instance:
uuid: "3f435d1173b64c3db8f9a6630b178dfe"
format_stamp: "Formatted at 2026-06-19 16:43:27 on dist-test-slave-wvz7"
server_key: "1bf0b52682bdc9d96572d176532b3e25"
server_key_iv: "80e024b74ded509bb083ba026a0d9987"
server_key_version: "encryptionkey@0"
I20260619 16:43:27.698000 2795 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887385903558-388-0/minicluster-data/master-0/wal/instance:
uuid: "3f435d1173b64c3db8f9a6630b178dfe"
format_stamp: "Formatted at 2026-06-19 16:43:27 on dist-test-slave-wvz7"
server_key: "1bf0b52682bdc9d96572d176532b3e25"
server_key_iv: "80e024b74ded509bb083ba026a0d9987"
server_key_version: "encryptionkey@0"
I20260619 16:43:27.702649 2795 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260619 16:43:27.705910 2810 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:43:27.707180 2795 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 16:43:27.707365 2795 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887385903558-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887385903558-388-0/minicluster-data/master-0/wal
uuid: "3f435d1173b64c3db8f9a6630b178dfe"
format_stamp: "Formatted at 2026-06-19 16:43:27 on dist-test-slave-wvz7"
server_key: "1bf0b52682bdc9d96572d176532b3e25"
server_key_iv: "80e024b74ded509bb083ba026a0d9987"
server_key_version: "encryptionkey@0"
I20260619 16:43:27.707532 2795 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887385903558-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887385903558-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887385903558-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:43:27.730158 2795 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:43:27.734040 2795 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:43:27.734287 2795 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:43:27.750329 2795 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:45577
I20260619 16:43:27.750371 2861 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:45577 every 8 connection(s)
I20260619 16:43:27.751566 2795 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887385903558-388-0/minicluster-data/master-0/data/info.pb
I20260619 16:43:27.754987 2862 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:43:27.756407 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu as pid 2795
I20260619 16:43:27.756577 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887385903558-388-0/minicluster-data/master-0/wal/instance
I20260619 16:43:27.757160 388 external_mini_cluster.cc:1428] Setting key 31da9f0ca897e3f34f58fb5c7901140f
I20260619 16:43:27.765852 2862 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3f435d1173b64c3db8f9a6630b178dfe: Bootstrap starting.
I20260619 16:43:27.769382 2862 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3f435d1173b64c3db8f9a6630b178dfe: Neither blocks nor log segments found. Creating new log.
I20260619 16:43:27.770577 2862 log.cc:824] T 00000000000000000000000000000000 P 3f435d1173b64c3db8f9a6630b178dfe: Log is configured to *not* fsync() on all Append() calls
Jun 19 16:43:27 dist-test-slave-wvz7 krb5kdc[2782](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887407, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 16:43:27.773291 2862 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3f435d1173b64c3db8f9a6630b178dfe: No bootstrap required, opened a new log
I20260619 16:43:27.778486 2865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:27.758989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53578 (local address 127.0.97.62:45577)
0619 16:43:27.759719 (+ 730us) server_negotiation.cc:206] Beginning negotiation
0619 16:43:27.759729 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 16:43:27.759766 (+ 37us) server_negotiation.cc:407] Connection header received
0619 16:43:27.761348 (+ 1582us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:43:27.761369 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:43:27.761727 (+ 358us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:43:27.762055 (+ 328us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:43:27.763735 (+ 1680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:27.764997 (+ 1262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:27.766649 (+ 1652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:27.767366 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:27.767500 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:43:27.771738 (+ 4238us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:43:27.771746 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:43:27.771760 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:43:27.771796 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:43:27.774704 (+ 2908us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:43:27.775340 (+ 636us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:43:27.775347 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:43:27.775353 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:43:27.775508 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:43:27.775857 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:43:27.775860 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:43:27.775862 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:43:27.776358 (+ 496us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:43:27.776613 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0619 16:43:27.777038 (+ 425us) server_negotiation.cc:299] Negotiation successful
0619 16:43:27.777436 (+ 398us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":522,"thread_start_us":306,"threads_started":1}
I20260619 16:43:27.780699 2862 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3f435d1173b64c3db8f9a6630b178dfe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f435d1173b64c3db8f9a6630b178dfe" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45577 } }
I20260619 16:43:27.781069 2862 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3f435d1173b64c3db8f9a6630b178dfe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:43:27.781160 2862 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3f435d1173b64c3db8f9a6630b178dfe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f435d1173b64c3db8f9a6630b178dfe, State: Initialized, Role: FOLLOWER
I20260619 16:43:27.781641 2862 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3f435d1173b64c3db8f9a6630b178dfe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f435d1173b64c3db8f9a6630b178dfe" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45577 } }
I20260619 16:43:27.781874 2862 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3f435d1173b64c3db8f9a6630b178dfe [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:43:27.781965 2862 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3f435d1173b64c3db8f9a6630b178dfe [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:43:27.782078 2862 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3f435d1173b64c3db8f9a6630b178dfe [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:43:27.784049 2862 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3f435d1173b64c3db8f9a6630b178dfe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f435d1173b64c3db8f9a6630b178dfe" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45577 } }
I20260619 16:43:27.784536 2862 leader_election.cc:302] T 00000000000000000000000000000000 P 3f435d1173b64c3db8f9a6630b178dfe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3f435d1173b64c3db8f9a6630b178dfe; no voters:
I20260619 16:43:27.785158 2862 leader_election.cc:288] T 00000000000000000000000000000000 P 3f435d1173b64c3db8f9a6630b178dfe [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 16:43:27.785252 2867 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3f435d1173b64c3db8f9a6630b178dfe [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:43:27.785893 2867 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3f435d1173b64c3db8f9a6630b178dfe [term 1 LEADER]: Becoming Leader. State: Replica: 3f435d1173b64c3db8f9a6630b178dfe, State: Running, Role: LEADER
I20260619 16:43:27.786468 2867 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3f435d1173b64c3db8f9a6630b178dfe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f435d1173b64c3db8f9a6630b178dfe" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45577 } }
I20260619 16:43:27.787284 2862 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3f435d1173b64c3db8f9a6630b178dfe [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:43:27.789240 2869 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3f435d1173b64c3db8f9a6630b178dfe [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3f435d1173b64c3db8f9a6630b178dfe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f435d1173b64c3db8f9a6630b178dfe" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45577 } } }
I20260619 16:43:27.789261 2868 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3f435d1173b64c3db8f9a6630b178dfe [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3f435d1173b64c3db8f9a6630b178dfe. Latest consensus state: current_term: 1 leader_uuid: "3f435d1173b64c3db8f9a6630b178dfe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f435d1173b64c3db8f9a6630b178dfe" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45577 } } }
I20260619 16:43:27.789628 2869 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3f435d1173b64c3db8f9a6630b178dfe [sys.catalog]: This master's current role is: LEADER
I20260619 16:43:27.789625 2868 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3f435d1173b64c3db8f9a6630b178dfe [sys.catalog]: This master's current role is: LEADER
I20260619 16:43:27.791186 2873 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:43:27.796324 2873 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:43:27.804929 2873 catalog_manager.cc:1347] Generated new cluster ID: bd7c0d9e397249bdb8e9718c7380c450
I20260619 16:43:27.805042 2873 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:43:27.822999 2873 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 16:43:27.824193 2873 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:43:27.830361 2873 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3f435d1173b64c3db8f9a6630b178dfe: Generated new TSK 0
I20260619 16:43:27.831197 2873 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887385903558-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887385903558-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260619 16:43:27.877218 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu
/tmp/dist-test-taskEs2EH8/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887385903558-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887385903558-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887385903558-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887385903558-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45577
--keytab_file=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887385903558-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36959
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887385903558-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887385903558-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887385903558-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:43:28.177748 2888 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:43:28.178005 2888 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:43:28.178104 2888 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:43:28.185032 2888 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:43:28.185128 2888 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:43:28.185243 2888 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260619 16:43:28.194581 2888 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36959
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887385903558-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887385903558-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887385903558-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887385903558-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45577
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887385903558-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:43:28.195726 2888 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:43:28.196918 2888 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:43:28.209022 2894 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:43:28.208941 2893 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:43:28.208992 2896 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:43:28.209846 2888 server_base.cc:1034] running on GCE node
I20260619 16:43:28.210466 2888 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:43:28.211459 2888 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:43:28.212662 2888 hybrid_clock.cc:648] HybridClock initialized: now 1781887408212604 us; error 55 us; skew 500 ppm
I20260619 16:43:28.212976 2888 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:43:28 dist-test-slave-wvz7 krb5kdc[2782](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887408, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:43:28.215934 2888 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260619 16:43:28.217294 2888 webserver.cc:460] Webserver started at http://127.0.97.1:36095/ using document root <none> and password file <none>
I20260619 16:43:28.217988 2888 fs_manager.cc:359] Metadata directory not provided
I20260619 16:43:28.218099 2888 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:43:28.218372 2888 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:43:28.221118 2888 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887385903558-388-0/minicluster-data/ts-0/data/instance:
uuid: "a8bd728c1dc040b6b06623df4af33a62"
format_stamp: "Formatted at 2026-06-19 16:43:28 on dist-test-slave-wvz7"
server_key: "22136366bc3b8f851927227167104b1e"
server_key_iv: "3b16f06a2a773f5ca80b771c1555f9fc"
server_key_version: "encryptionkey@0"
I20260619 16:43:28.221854 2888 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887385903558-388-0/minicluster-data/ts-0/wal/instance:
uuid: "a8bd728c1dc040b6b06623df4af33a62"
format_stamp: "Formatted at 2026-06-19 16:43:28 on dist-test-slave-wvz7"
server_key: "22136366bc3b8f851927227167104b1e"
server_key_iv: "3b16f06a2a773f5ca80b771c1555f9fc"
server_key_version: "encryptionkey@0"
I20260619 16:43:28.226647 2888 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260619 16:43:28.229902 2903 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:43:28.231146 2888 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260619 16:43:28.231287 2888 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887385903558-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887385903558-388-0/minicluster-data/ts-0/wal
uuid: "a8bd728c1dc040b6b06623df4af33a62"
format_stamp: "Formatted at 2026-06-19 16:43:28 on dist-test-slave-wvz7"
server_key: "22136366bc3b8f851927227167104b1e"
server_key_iv: "3b16f06a2a773f5ca80b771c1555f9fc"
server_key_version: "encryptionkey@0"
I20260619 16:43:28.231447 2888 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887385903558-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887385903558-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887385903558-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:43:28.249104 2888 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:43:28.252940 2888 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:43:28.253180 2888 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:43:28.254057 2888 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:43:28.255370 2888 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:43:28.255445 2888 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:43:28.255561 2888 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:43:28.255609 2888 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:43:28.283829 2888 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:38937
I20260619 16:43:28.283872 3015 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:38937 every 8 connection(s)
I20260619 16:43:28.285024 2888 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887385903558-388-0/minicluster-data/ts-0/data/info.pb
I20260619 16:43:28.293323 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu as pid 2888
I20260619 16:43:28.293502 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887385903558-388-0/minicluster-data/ts-0/wal/instance
I20260619 16:43:28.294099 388 external_mini_cluster.cc:1428] Setting key 0839494c9611a5af330d085b4d3a6134
Jun 19 16:43:28 dist-test-slave-wvz7 krb5kdc[2782](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887408, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 16:43:28.304531 3018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:28.287392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:50991 (local address 127.0.97.62:45577)
0619 16:43:28.287814 (+ 422us) server_negotiation.cc:206] Beginning negotiation
0619 16:43:28.287818 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:43:28.288848 (+ 1030us) server_negotiation.cc:407] Connection header received
0619 16:43:28.290613 (+ 1765us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:43:28.290618 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:43:28.290697 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:43:28.290791 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:43:28.292312 (+ 1521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:28.293002 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:28.294643 (+ 1641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:28.295077 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:28.295174 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:43:28.298536 (+ 3362us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:43:28.298541 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:43:28.298545 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:43:28.298580 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:43:28.301527 (+ 2947us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:43:28.302328 (+ 801us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:43:28.302334 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:43:28.302337 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:43:28.302443 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:43:28.302869 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:43:28.302874 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:43:28.302894 (+ 20us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:43:28.303197 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:43:28.303339 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0619 16:43:28.303974 (+ 635us) server_negotiation.cc:299] Negotiation successful
0619 16:43:28.304180 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":319,"thread_start_us":205,"threads_started":1}
I20260619 16:43:28.305541 3019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:28.287969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45577 (local address 127.0.97.1:50991)
0619 16:43:28.288679 (+ 710us) negotiation.cc:107] Waiting for socket to connect
0619 16:43:28.288741 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0619 16:43:28.290359 (+ 1618us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:43:28.291054 (+ 695us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:43:28.291062 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:43:28.291541 (+ 479us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:43:28.292118 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:43:28.292153 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:43:28.293148 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:43:28.293152 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:43:28.294512 (+ 1360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:43:28.294524 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:43:28.295279 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:43:28.295286 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:43:28.295550 (+ 264us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:43:28.296277 (+ 727us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:43:28.296302 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:43:28.298333 (+ 2031us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:43:28.301658 (+ 3325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:43:28.301667 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:43:28.301699 (+ 32us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:43:28.302212 (+ 513us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:43:28.302536 (+ 324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:43:28.302541 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:43:28.302546 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:43:28.302771 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:43:28.303315 (+ 544us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:43:28.303323 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:43:28.303638 (+ 315us) client_negotiation.cc:769] Sending connection context
0619 16:43:28.303874 (+ 236us) client_negotiation.cc:240] Negotiation successful
0619 16:43:28.304365 (+ 491us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":479,"thread_start_us":304,"threads_started":1}
I20260619 16:43:28.307021 3016 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45577
I20260619 16:43:28.307286 3016 heartbeater.cc:464] Registering TS with master...
I20260619 16:43:28.307960 3016 heartbeater.cc:511] Master 127.0.97.62:45577 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260619 16:43:28.310205 2827 ts_manager.cc:194] Registered new tserver with Master: a8bd728c1dc040b6b06623df4af33a62 (127.0.97.1:38937)
I20260619 16:43:28.311692 2827 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:50991
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887385903558-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887385903558-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260619 16:43:28.333619 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu
/tmp/dist-test-taskEs2EH8/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887385903558-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887385903558-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887385903558-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887385903558-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45577
--keytab_file=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887385903558-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36959
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887385903558-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887385903558-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887385903558-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:43:28.632946 3022 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:43:28.633224 3022 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:43:28.633332 3022 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:43:28.640836 3022 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:43:28.640939 3022 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:43:28.641081 3022 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260619 16:43:28.650641 3022 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36959
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887385903558-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887385903558-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887385903558-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887385903558-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45577
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887385903558-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:43:28.651813 3022 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:43:28.653025 3022 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:43:28.665323 3028 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:43:28.665295 3027 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:43:28.665371 3030 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:43:28.666154 3022 server_base.cc:1034] running on GCE node
I20260619 16:43:28.666818 3022 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:43:28.667919 3022 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:43:28.669095 3022 hybrid_clock.cc:648] HybridClock initialized: now 1781887408669025 us; error 63 us; skew 500 ppm
I20260619 16:43:28.669374 3022 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:43:28 dist-test-slave-wvz7 krb5kdc[2782](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887408, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:43:28.672223 3022 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260619 16:43:28.673514 3022 webserver.cc:460] Webserver started at http://127.0.97.2:46273/ using document root <none> and password file <none>
I20260619 16:43:28.674199 3022 fs_manager.cc:359] Metadata directory not provided
I20260619 16:43:28.674302 3022 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:43:28.674582 3022 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:43:28.677322 3022 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887385903558-388-0/minicluster-data/ts-1/data/instance:
uuid: "2207dc2354144a8f8ec523c2f345bcfb"
format_stamp: "Formatted at 2026-06-19 16:43:28 on dist-test-slave-wvz7"
server_key: "c48658be7f46a4bd74838c54d1f98935"
server_key_iv: "d79959ada739d235e27127e954a1ba83"
server_key_version: "encryptionkey@0"
I20260619 16:43:28.678062 3022 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887385903558-388-0/minicluster-data/ts-1/wal/instance:
uuid: "2207dc2354144a8f8ec523c2f345bcfb"
format_stamp: "Formatted at 2026-06-19 16:43:28 on dist-test-slave-wvz7"
server_key: "c48658be7f46a4bd74838c54d1f98935"
server_key_iv: "d79959ada739d235e27127e954a1ba83"
server_key_version: "encryptionkey@0"
I20260619 16:43:28.682868 3022 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260619 16:43:28.686250 3037 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:43:28.687553 3022 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260619 16:43:28.687711 3022 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887385903558-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887385903558-388-0/minicluster-data/ts-1/wal
uuid: "2207dc2354144a8f8ec523c2f345bcfb"
format_stamp: "Formatted at 2026-06-19 16:43:28 on dist-test-slave-wvz7"
server_key: "c48658be7f46a4bd74838c54d1f98935"
server_key_iv: "d79959ada739d235e27127e954a1ba83"
server_key_version: "encryptionkey@0"
I20260619 16:43:28.687870 3022 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887385903558-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887385903558-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887385903558-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:43:28.713855 3022 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:43:28.717895 3022 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:43:28.718154 3022 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:43:28.719048 3022 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:43:28.720304 3022 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:43:28.720371 3022 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:43:28.720474 3022 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:43:28.720520 3022 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:43:28.748370 3022 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:46367
I20260619 16:43:28.748422 3149 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:46367 every 8 connection(s)
I20260619 16:43:28.749531 3022 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887385903558-388-0/minicluster-data/ts-1/data/info.pb
I20260619 16:43:28.759028 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu as pid 3022
I20260619 16:43:28.759213 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887385903558-388-0/minicluster-data/ts-1/wal/instance
I20260619 16:43:28.759639 388 external_mini_cluster.cc:1428] Setting key eeac7294556c8e975ea9a67efbd3a31f
Jun 19 16:43:28 dist-test-slave-wvz7 krb5kdc[2782](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887408, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 16:43:28.771987 3018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:28.751997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:50527 (local address 127.0.97.62:45577)
0619 16:43:28.752184 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0619 16:43:28.752190 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:43:28.753451 (+ 1261us) server_negotiation.cc:407] Connection header received
0619 16:43:28.755961 (+ 2510us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:43:28.755965 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:43:28.756033 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:43:28.756142 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:43:28.758000 (+ 1858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:28.758771 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:28.760957 (+ 2186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:28.761564 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:28.761709 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:43:28.765828 (+ 4119us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:43:28.765835 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:43:28.765840 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:43:28.765877 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:43:28.768852 (+ 2975us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:43:28.769607 (+ 755us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:43:28.769612 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:43:28.769614 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:43:28.769812 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:43:28.770248 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:43:28.770254 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:43:28.770256 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:43:28.770536 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:43:28.770675 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0619 16:43:28.771389 (+ 714us) server_negotiation.cc:299] Negotiation successful
0619 16:43:28.771623 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260619 16:43:28.773000 3152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:28.752509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45577 (local address 127.0.97.2:50527)
0619 16:43:28.753270 (+ 761us) negotiation.cc:107] Waiting for socket to connect
0619 16:43:28.753325 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0619 16:43:28.755654 (+ 2329us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:43:28.756369 (+ 715us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:43:28.756380 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:43:28.757049 (+ 669us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:43:28.757807 (+ 758us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:43:28.757833 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:43:28.758925 (+ 1092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:43:28.758931 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:43:28.760813 (+ 1882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:43:28.760827 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:43:28.761818 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:43:28.761825 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:43:28.762112 (+ 287us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:43:28.762829 (+ 717us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:43:28.762856 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:43:28.765552 (+ 2696us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:43:28.769001 (+ 3449us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:43:28.769010 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:43:28.769034 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:43:28.769472 (+ 438us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:43:28.769915 (+ 443us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:43:28.769922 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:43:28.769926 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:43:28.770126 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:43:28.770683 (+ 557us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:43:28.770692 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:43:28.771017 (+ 325us) client_negotiation.cc:769] Sending connection context
0619 16:43:28.771277 (+ 260us) client_negotiation.cc:240] Negotiation successful
0619 16:43:28.771787 (+ 510us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":495,"thread_start_us":281,"threads_started":1}
I20260619 16:43:28.774663 3150 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45577
I20260619 16:43:28.774961 3150 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260619 16:43:28.775664 3150 heartbeater.cc:511] Master 127.0.97.62:45577 requested a full tablet report, sending...
I20260619 16:43:28.776969 2827 ts_manager.cc:194] Registered new tserver with Master: 2207dc2354144a8f8ec523c2f345bcfb (127.0.97.2:46367)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
I20260619 16:43:28.777971 2827 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:50527
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887385903558-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887385903558-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260619 16:43:28.798538 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu
/tmp/dist-test-taskEs2EH8/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887385903558-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887385903558-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887385903558-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887385903558-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45577
--keytab_file=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887385903558-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36959
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887385903558-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887385903558-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887385903558-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:43:29.100229 3155 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:43:29.100500 3155 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:43:29.100632 3155 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:43:29.107669 3155 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:43:29.107784 3155 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:43:29.107928 3155 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260619 16:43:29.117265 3155 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36959
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887385903558-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887385903558-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887385903558-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887385903558-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45577
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887385903558-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:43:29.118476 3155 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:43:29.119745 3155 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:43:29.131374 3160 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:43:29.131412 3163 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:43:29.131424 3161 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:43:29.132417 3155 server_base.cc:1034] running on GCE node
I20260619 16:43:29.132993 3155 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:43:29.133900 3155 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:43:29.135082 3155 hybrid_clock.cc:648] HybridClock initialized: now 1781887409135040 us; error 41 us; skew 500 ppm
I20260619 16:43:29.135367 3155 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:43:29 dist-test-slave-wvz7 krb5kdc[2782](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887409, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:43:29.138394 3155 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260619 16:43:29.139761 3155 webserver.cc:460] Webserver started at http://127.0.97.3:35105/ using document root <none> and password file <none>
I20260619 16:43:29.140385 3155 fs_manager.cc:359] Metadata directory not provided
I20260619 16:43:29.140498 3155 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:43:29.140779 3155 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:43:29.143563 3155 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887385903558-388-0/minicluster-data/ts-2/data/instance:
uuid: "009adfe0a1974acf834f0a645eb4622d"
format_stamp: "Formatted at 2026-06-19 16:43:29 on dist-test-slave-wvz7"
server_key: "22844971cac897437252d25c5b3cacab"
server_key_iv: "4c79833acae86eb309a78cb78900b83c"
server_key_version: "encryptionkey@0"
I20260619 16:43:29.144289 3155 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887385903558-388-0/minicluster-data/ts-2/wal/instance:
uuid: "009adfe0a1974acf834f0a645eb4622d"
format_stamp: "Formatted at 2026-06-19 16:43:29 on dist-test-slave-wvz7"
server_key: "22844971cac897437252d25c5b3cacab"
server_key_iv: "4c79833acae86eb309a78cb78900b83c"
server_key_version: "encryptionkey@0"
I20260619 16:43:29.149094 3155 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260619 16:43:29.152336 3170 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:43:29.153545 3155 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 16:43:29.153692 3155 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887385903558-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887385903558-388-0/minicluster-data/ts-2/wal
uuid: "009adfe0a1974acf834f0a645eb4622d"
format_stamp: "Formatted at 2026-06-19 16:43:29 on dist-test-slave-wvz7"
server_key: "22844971cac897437252d25c5b3cacab"
server_key_iv: "4c79833acae86eb309a78cb78900b83c"
server_key_version: "encryptionkey@0"
I20260619 16:43:29.153894 3155 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887385903558-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887385903558-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887385903558-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:43:29.186790 3155 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:43:29.191139 3155 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:43:29.191413 3155 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:43:29.192473 3155 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:43:29.194061 3155 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:43:29.194139 3155 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:43:29.194262 3155 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:43:29.194311 3155 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:43:29.222970 3155 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:41065
I20260619 16:43:29.223026 3282 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:41065 every 8 connection(s)
I20260619 16:43:29.224211 3155 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887385903558-388-0/minicluster-data/ts-2/data/info.pb
I20260619 16:43:29.234133 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu as pid 3155
I20260619 16:43:29.234349 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887385903558-388-0/minicluster-data/ts-2/wal/instance
I20260619 16:43:29.234853 388 external_mini_cluster.cc:1428] Setting key 08ae635be0e2bd695878f87671168681
Jun 19 16:43:29 dist-test-slave-wvz7 krb5kdc[2782](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887409, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 16:43:29.243196 3018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:29.226556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:58247 (local address 127.0.97.62:45577)
0619 16:43:29.226769 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0619 16:43:29.226776 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:43:29.228107 (+ 1331us) server_negotiation.cc:407] Connection header received
0619 16:43:29.229804 (+ 1697us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:43:29.229808 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:43:29.229873 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:43:29.229956 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:43:29.231338 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:29.232056 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:29.233558 (+ 1502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:29.234118 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:29.234257 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:43:29.237861 (+ 3604us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:43:29.237869 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:43:29.237874 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:43:29.237917 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:43:29.240394 (+ 2477us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:43:29.241060 (+ 666us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:43:29.241066 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:43:29.241069 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:43:29.241196 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:43:29.241568 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:43:29.241582 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:43:29.241585 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:43:29.241885 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:43:29.242009 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0619 16:43:29.242662 (+ 653us) server_negotiation.cc:299] Negotiation successful
0619 16:43:29.242873 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260619 16:43:29.244107 3285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:29.227263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45577 (local address 127.0.97.3:58247)
0619 16:43:29.227933 (+ 670us) negotiation.cc:107] Waiting for socket to connect
0619 16:43:29.227976 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0619 16:43:29.229533 (+ 1557us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:43:29.230155 (+ 622us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:43:29.230164 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:43:29.230639 (+ 475us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:43:29.231174 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:43:29.231191 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:43:29.232198 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:43:29.232201 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:43:29.233418 (+ 1217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:43:29.233429 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:43:29.234362 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:43:29.234369 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:43:29.234594 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:43:29.235281 (+ 687us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:43:29.235307 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:43:29.237654 (+ 2347us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:43:29.240516 (+ 2862us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:43:29.240525 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:43:29.240540 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:43:29.240945 (+ 405us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:43:29.241292 (+ 347us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:43:29.241297 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:43:29.241301 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:43:29.241470 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:43:29.241995 (+ 525us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:43:29.242002 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:43:29.242320 (+ 318us) client_negotiation.cc:769] Sending connection context
0619 16:43:29.242552 (+ 232us) client_negotiation.cc:240] Negotiation successful
0619 16:43:29.243041 (+ 489us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":451,"thread_start_us":275,"threads_started":1}
I20260619 16:43:29.245365 3283 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45577
I20260619 16:43:29.245639 3283 heartbeater.cc:464] Registering TS with master...
I20260619 16:43:29.246372 3283 heartbeater.cc:511] Master 127.0.97.62:45577 requested a full tablet report, sending...
I20260619 16:43:29.247782 2827 ts_manager.cc:194] Registered new tserver with Master: 009adfe0a1974acf834f0a645eb4622d (127.0.97.3:41065)
I20260619 16:43:29.248797 2827 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:58247
I20260619 16:43:29.249871 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260619 16:43:29.275339 3018 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:43:29.273129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53582 (local address 127.0.97.62:45577)
0619 16:43:29.273355 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0619 16:43:29.273362 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:43:29.273676 (+ 314us) server_negotiation.cc:407] Connection header received
0619 16:43:29.273856 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:43:29.273860 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:43:29.273924 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:43:29.274002 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:43:29.275086 (+ 1084us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53582: BlockingRecv error: recv got EOF from 127.0.0.1:53582 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
I20260619 16:43:29.278687 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu pid 2888
W20260619 16:43:29.281294 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu pid 2888: Not authorized: Client connection negotiation failed: client connection to 127.0.97.1:38937: server requires authentication, but client does not have Kerberos credentials available
I20260619 16:43:29.281438 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu with pid 2888
I20260619 16:43:29.293107 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu pid 3022
W20260619 16:43:29.295563 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu pid 3022: Not authorized: Client connection negotiation failed: client connection to 127.0.97.2:46367: server requires authentication, but client does not have Kerberos credentials available
I20260619 16:43:29.295714 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu with pid 3022
I20260619 16:43:29.308938 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu pid 3155
W20260619 16:43:29.312848 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu pid 3155: Not authorized: Client connection negotiation failed: client connection to 127.0.97.3:41065: server requires authentication, but client does not have Kerberos credentials available
I20260619 16:43:29.313006 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu with pid 3155
I20260619 16:43:29.324311 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu pid 2795
I20260619 16:43:29.325373 2857 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:43:29.424324 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu with pid 2795
2026-06-19T16:43:29Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (3234 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781887385903558-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 16:43:29 dist-test-slave-wvz7 krb5kdc[3300](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 16:43:29 dist-test-slave-wvz7 krb5kdc[3300](info): set up 2 sockets
Jun 19 16:43:29 dist-test-slave-wvz7 krb5kdc[3300](info): commencing operation
krb5kdc: starting...
W20260619 16:43:30.413698 388 mini_kdc.cc:121] Time spent starting KDC: real 0.953s user 0.001s sys 0.014s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781887385903558-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781887385903558-388-0/krb5kdc/test-user.keytab.
Jun 19 16:43:30 dist-test-slave-wvz7 krb5kdc[3300](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887410, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T16:43:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T16:43:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781887385903558-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781887385903558-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260619 16:43:30.581435 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu
/tmp/dist-test-taskEs2EH8/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781887385903558-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781887385903558-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781887385903558-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781887385903558-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:44427
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781887385903558-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33535
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:44427
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781887385903558-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781887385903558-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781887385903558-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:43:30.880124 3313 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:43:30.880420 3313 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:43:30.880548 3313 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:43:30.887903 3313 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:43:30.888018 3313 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:43:30.888098 3313 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:43:30.888176 3313 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:43:30.888244 3313 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:43:30.898170 3313 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33535
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781887385903558-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781887385903558-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:44427
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781887385903558-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:44427
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781887385903558-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781887385903558-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:43:30.899372 3313 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:43:30.900614 3313 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:43:30.911459 3321 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:43:30.911569 3319 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:43:30.911571 3318 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:43:30.912324 3313 server_base.cc:1034] running on GCE node
I20260619 16:43:30.913002 3313 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:43:30.914311 3313 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:43:30.915493 3313 hybrid_clock.cc:648] HybridClock initialized: now 1781887410915457 us; error 30 us; skew 500 ppm
I20260619 16:43:30.915796 3313 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:43:30 dist-test-slave-wvz7 krb5kdc[3300](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887410, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:43:30.922834 3313 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260619 16:43:30.924135 3313 webserver.cc:460] Webserver started at http://127.0.97.62:45597/ using document root <none> and password file <none>
I20260619 16:43:30.924724 3313 fs_manager.cc:359] Metadata directory not provided
I20260619 16:43:30.924813 3313 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:43:30.925037 3313 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:43:30.927930 3313 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781887385903558-388-0/minicluster-data/master-0/data/instance:
uuid: "5957d0eef8f14fd495b49d28e1f2a8a3"
format_stamp: "Formatted at 2026-06-19 16:43:30 on dist-test-slave-wvz7"
server_key: "c456db84b65d0f41dc9b7051d636ec6f"
server_key_iv: "af70a79e1e6eba54bb87bdd2e100cb8f"
server_key_version: "encryptionkey@0"
I20260619 16:43:30.928658 3313 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781887385903558-388-0/minicluster-data/master-0/wal/instance:
uuid: "5957d0eef8f14fd495b49d28e1f2a8a3"
format_stamp: "Formatted at 2026-06-19 16:43:30 on dist-test-slave-wvz7"
server_key: "c456db84b65d0f41dc9b7051d636ec6f"
server_key_iv: "af70a79e1e6eba54bb87bdd2e100cb8f"
server_key_version: "encryptionkey@0"
I20260619 16:43:30.933327 3313 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260619 16:43:30.936497 3328 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:43:30.937836 3313 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260619 16:43:30.937978 3313 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781887385903558-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781887385903558-388-0/minicluster-data/master-0/wal
uuid: "5957d0eef8f14fd495b49d28e1f2a8a3"
format_stamp: "Formatted at 2026-06-19 16:43:30 on dist-test-slave-wvz7"
server_key: "c456db84b65d0f41dc9b7051d636ec6f"
server_key_iv: "af70a79e1e6eba54bb87bdd2e100cb8f"
server_key_version: "encryptionkey@0"
I20260619 16:43:30.938143 3313 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781887385903558-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781887385903558-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781887385903558-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:43:30.949807 3313 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:43:30.953397 3313 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:43:30.953629 3313 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:43:30.969964 3313 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:44427
I20260619 16:43:30.970007 3379 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:44427 every 8 connection(s)
I20260619 16:43:30.971184 3313 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781887385903558-388-0/minicluster-data/master-0/data/info.pb
I20260619 16:43:30.974809 3380 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:43:30.976854 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu as pid 3313
I20260619 16:43:30.977084 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781887385903558-388-0/minicluster-data/master-0/wal/instance
I20260619 16:43:30.977921 388 external_mini_cluster.cc:1428] Setting key ee7cf1ae9c77256bf6b15a7bfc1cc645
I20260619 16:43:30.986395 3380 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5957d0eef8f14fd495b49d28e1f2a8a3: Bootstrap starting.
I20260619 16:43:30.989499 3380 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5957d0eef8f14fd495b49d28e1f2a8a3: Neither blocks nor log segments found. Creating new log.
Jun 19 16:43:30 dist-test-slave-wvz7 krb5kdc[3300](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887410, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 16:43:30.990528 3380 log.cc:824] T 00000000000000000000000000000000 P 5957d0eef8f14fd495b49d28e1f2a8a3: Log is configured to *not* fsync() on all Append() calls
I20260619 16:43:30.993585 3380 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5957d0eef8f14fd495b49d28e1f2a8a3: No bootstrap required, opened a new log
I20260619 16:43:30.999423 3383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:30.979830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40596 (local address 127.0.97.62:44427)
0619 16:43:30.980583 (+ 753us) server_negotiation.cc:206] Beginning negotiation
0619 16:43:30.980598 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 16:43:30.980655 (+ 57us) server_negotiation.cc:407] Connection header received
0619 16:43:30.982284 (+ 1629us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:43:30.982305 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:43:30.982671 (+ 366us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:43:30.982984 (+ 313us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:43:30.984328 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:30.985576 (+ 1248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:30.986848 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:30.987468 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:30.987638 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:43:30.991316 (+ 3678us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:43:30.991324 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:43:30.991337 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:43:30.991386 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:43:30.995067 (+ 3681us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:43:30.995669 (+ 602us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:43:30.995677 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:43:30.995685 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:43:30.995890 (+ 205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:43:30.996229 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:43:30.996235 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:43:30.996238 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:43:30.996957 (+ 719us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:43:30.997279 (+ 322us) server_negotiation.cc:1035] Waiting for connection context
0619 16:43:30.997589 (+ 310us) server_negotiation.cc:299] Negotiation successful
0619 16:43:30.998056 (+ 467us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":560,"thread_start_us":343,"threads_started":1}
I20260619 16:43:31.003487 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 16:43:31.003815 3380 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5957d0eef8f14fd495b49d28e1f2a8a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5957d0eef8f14fd495b49d28e1f2a8a3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44427 } }
I20260619 16:43:31.004163 3380 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5957d0eef8f14fd495b49d28e1f2a8a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:43:31.004266 3380 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5957d0eef8f14fd495b49d28e1f2a8a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5957d0eef8f14fd495b49d28e1f2a8a3, State: Initialized, Role: FOLLOWER
I20260619 16:43:31.004851 3380 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5957d0eef8f14fd495b49d28e1f2a8a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5957d0eef8f14fd495b49d28e1f2a8a3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44427 } }
I20260619 16:43:31.005074 3380 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5957d0eef8f14fd495b49d28e1f2a8a3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:43:31.005203 3380 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5957d0eef8f14fd495b49d28e1f2a8a3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:43:31.005363 3380 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5957d0eef8f14fd495b49d28e1f2a8a3 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:43:31.007575 3380 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5957d0eef8f14fd495b49d28e1f2a8a3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5957d0eef8f14fd495b49d28e1f2a8a3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44427 } }
I20260619 16:43:31.008049 3380 leader_election.cc:302] T 00000000000000000000000000000000 P 5957d0eef8f14fd495b49d28e1f2a8a3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5957d0eef8f14fd495b49d28e1f2a8a3; no voters:
I20260619 16:43:31.008626 3380 leader_election.cc:288] T 00000000000000000000000000000000 P 5957d0eef8f14fd495b49d28e1f2a8a3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 16:43:31.008775 3385 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5957d0eef8f14fd495b49d28e1f2a8a3 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:43:31.009965 3385 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5957d0eef8f14fd495b49d28e1f2a8a3 [term 1 LEADER]: Becoming Leader. State: Replica: 5957d0eef8f14fd495b49d28e1f2a8a3, State: Running, Role: LEADER
I20260619 16:43:31.010426 3385 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5957d0eef8f14fd495b49d28e1f2a8a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5957d0eef8f14fd495b49d28e1f2a8a3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44427 } }
I20260619 16:43:31.010916 3380 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5957d0eef8f14fd495b49d28e1f2a8a3 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:43:31.013219 3387 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5957d0eef8f14fd495b49d28e1f2a8a3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5957d0eef8f14fd495b49d28e1f2a8a3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5957d0eef8f14fd495b49d28e1f2a8a3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44427 } } }
I20260619 16:43:31.013571 3387 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5957d0eef8f14fd495b49d28e1f2a8a3 [sys.catalog]: This master's current role is: LEADER
I20260619 16:43:31.018260 3388 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5957d0eef8f14fd495b49d28e1f2a8a3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5957d0eef8f14fd495b49d28e1f2a8a3. Latest consensus state: current_term: 1 leader_uuid: "5957d0eef8f14fd495b49d28e1f2a8a3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5957d0eef8f14fd495b49d28e1f2a8a3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44427 } } }
I20260619 16:43:31.019398 3388 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5957d0eef8f14fd495b49d28e1f2a8a3 [sys.catalog]: This master's current role is: LEADER
I20260619 16:43:31.019956 3392 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:43:31.024827 3392 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:43:31.051529 3392 catalog_manager.cc:1347] Generated new cluster ID: 5ee2d0063f0f4b6aa21052afa02d468a
I20260619 16:43:31.051678 3392 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:43:31.070412 3392 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 16:43:31.072427 3392 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:43:31.088524 3392 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5957d0eef8f14fd495b49d28e1f2a8a3: Generated new TSK 0
I20260619 16:43:31.089643 3392 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 16:43:31.096095 3383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:31.087504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40610 (local address 127.0.97.62:44427)
0619 16:43:31.087676 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0619 16:43:31.087683 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:43:31.088299 (+ 616us) server_negotiation.cc:407] Connection header received
0619 16:43:31.088576 (+ 277us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:43:31.088582 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:43:31.088714 (+ 132us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:43:31.088834 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:43:31.089997 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:31.091161 (+ 1164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:31.092421 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:31.093060 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:31.093192 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:43:31.094355 (+ 1163us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:43:31.094361 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:43:31.094366 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:43:31.095255 (+ 889us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:43:31.095330 (+ 75us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:43:31.095364 (+ 34us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:43:31.095470 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0619 16:43:31.095630 (+ 160us) server_negotiation.cc:299] Negotiation successful
0619 16:43:31.095723 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260619 16:43:31.103226 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu pid 3313
I20260619 16:43:31.104409 3375 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:43:31.193890 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu with pid 3313
2026-06-19T16:43:31Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1764 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887385903558-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 16:43:31 dist-test-slave-wvz7 krb5kdc[3414](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 16:43:31 dist-test-slave-wvz7 krb5kdc[3414](info): set up 2 sockets
Jun 19 16:43:31 dist-test-slave-wvz7 krb5kdc[3414](info): commencing operation
krb5kdc: starting...
W20260619 16:43:32.237663 388 mini_kdc.cc:121] Time spent starting KDC: real 1.013s user 0.000s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887385903558-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887385903558-388-0/krb5kdc/test-user.keytab.
Jun 19 16:43:32 dist-test-slave-wvz7 krb5kdc[3414](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887412, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T16:43:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T16:43:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887385903558-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887385903558-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260619 16:43:32.408277 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu
/tmp/dist-test-taskEs2EH8/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887385903558-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887385903558-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887385903558-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887385903558-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:41885
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887385903558-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41687
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:41885
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887385903558-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887385903558-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887385903558-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:43:32.708886 3427 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:43:32.709157 3427 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:43:32.709263 3427 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:43:32.716229 3427 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:43:32.716324 3427 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:43:32.716377 3427 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:43:32.716428 3427 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:43:32.716470 3427 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:43:32.725790 3427 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41687
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887385903558-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887385903558-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:41885
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887385903558-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:41885
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887385903558-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887385903558-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:43:32.726956 3427 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:43:32.728133 3427 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:43:32.738651 3432 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:43:32.738617 3433 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:43:32.738835 3435 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:43:32.739322 3427 server_base.cc:1034] running on GCE node
I20260619 16:43:32.740005 3427 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:43:32.741290 3427 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:43:32.742520 3427 hybrid_clock.cc:648] HybridClock initialized: now 1781887412742463 us; error 41 us; skew 500 ppm
I20260619 16:43:32.742836 3427 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:43:32 dist-test-slave-wvz7 krb5kdc[3414](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887412, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:43:32.745668 3427 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260619 16:43:32.746953 3427 webserver.cc:460] Webserver started at http://127.0.97.62:45625/ using document root <none> and password file <none>
I20260619 16:43:32.747550 3427 fs_manager.cc:359] Metadata directory not provided
I20260619 16:43:32.747639 3427 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:43:32.747865 3427 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:43:32.750698 3427 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887385903558-388-0/minicluster-data/master-0/data/instance:
uuid: "01db4b7096f343038b0995c1a919b60d"
format_stamp: "Formatted at 2026-06-19 16:43:32 on dist-test-slave-wvz7"
server_key: "a7cca0600503000b5cf6d456d5040184"
server_key_iv: "715164e5da57b90eeec439344e80c4c4"
server_key_version: "encryptionkey@0"
I20260619 16:43:32.751346 3427 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887385903558-388-0/minicluster-data/master-0/wal/instance:
uuid: "01db4b7096f343038b0995c1a919b60d"
format_stamp: "Formatted at 2026-06-19 16:43:32 on dist-test-slave-wvz7"
server_key: "a7cca0600503000b5cf6d456d5040184"
server_key_iv: "715164e5da57b90eeec439344e80c4c4"
server_key_version: "encryptionkey@0"
I20260619 16:43:32.756098 3427 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 16:43:32.759367 3442 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:43:32.760725 3427 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 16:43:32.760908 3427 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887385903558-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887385903558-388-0/minicluster-data/master-0/wal
uuid: "01db4b7096f343038b0995c1a919b60d"
format_stamp: "Formatted at 2026-06-19 16:43:32 on dist-test-slave-wvz7"
server_key: "a7cca0600503000b5cf6d456d5040184"
server_key_iv: "715164e5da57b90eeec439344e80c4c4"
server_key_version: "encryptionkey@0"
I20260619 16:43:32.761104 3427 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887385903558-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887385903558-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887385903558-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:43:32.776336 3427 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:43:32.780048 3427 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:43:32.780297 3427 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:43:32.795433 3427 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:41885
I20260619 16:43:32.795464 3493 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:41885 every 8 connection(s)
I20260619 16:43:32.796674 3427 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887385903558-388-0/minicluster-data/master-0/data/info.pb
I20260619 16:43:32.800100 3494 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:43:32.803294 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu as pid 3427
I20260619 16:43:32.803476 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887385903558-388-0/minicluster-data/master-0/wal/instance
I20260619 16:43:32.804220 388 external_mini_cluster.cc:1428] Setting key 8de68a4a2f292a2176dcfe7cff2e2bae
I20260619 16:43:32.812115 3494 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 01db4b7096f343038b0995c1a919b60d: Bootstrap starting.
I20260619 16:43:32.815601 3494 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 01db4b7096f343038b0995c1a919b60d: Neither blocks nor log segments found. Creating new log.
I20260619 16:43:32.816830 3494 log.cc:824] T 00000000000000000000000000000000 P 01db4b7096f343038b0995c1a919b60d: Log is configured to *not* fsync() on all Append() calls
Jun 19 16:43:32 dist-test-slave-wvz7 krb5kdc[3414](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887412, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 16:43:32.819502 3494 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 01db4b7096f343038b0995c1a919b60d: No bootstrap required, opened a new log
I20260619 16:43:32.827744 3497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:32.806144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48348 (local address 127.0.97.62:41885)
0619 16:43:32.806845 (+ 701us) server_negotiation.cc:206] Beginning negotiation
0619 16:43:32.806856 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 16:43:32.806894 (+ 38us) server_negotiation.cc:407] Connection header received
0619 16:43:32.808372 (+ 1478us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:43:32.808392 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:43:32.808769 (+ 377us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:43:32.809077 (+ 308us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:43:32.810682 (+ 1605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:32.811924 (+ 1242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:32.813566 (+ 1642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:32.814286 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:32.814424 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:43:32.818622 (+ 4198us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:43:32.818629 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:43:32.818646 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:43:32.818682 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:43:32.822244 (+ 3562us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:43:32.823084 (+ 840us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:43:32.823094 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:43:32.823102 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:43:32.823314 (+ 212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:43:32.823771 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:43:32.823776 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:43:32.823779 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:43:32.824458 (+ 679us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:43:32.825680 (+ 1222us) server_negotiation.cc:1035] Waiting for connection context
0619 16:43:32.825969 (+ 289us) server_negotiation.cc:299] Negotiation successful
0619 16:43:32.826405 (+ 436us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":513,"thread_start_us":331,"threads_started":1}
I20260619 16:43:32.829437 3494 raft_consensus.cc:348] T 00000000000000000000000000000000 P 01db4b7096f343038b0995c1a919b60d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01db4b7096f343038b0995c1a919b60d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41885 } }
I20260619 16:43:32.829823 3494 raft_consensus.cc:374] T 00000000000000000000000000000000 P 01db4b7096f343038b0995c1a919b60d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:43:32.829926 3494 raft_consensus.cc:729] T 00000000000000000000000000000000 P 01db4b7096f343038b0995c1a919b60d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 01db4b7096f343038b0995c1a919b60d, State: Initialized, Role: FOLLOWER
I20260619 16:43:32.830478 3494 consensus_queue.cc:260] T 00000000000000000000000000000000 P 01db4b7096f343038b0995c1a919b60d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01db4b7096f343038b0995c1a919b60d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41885 } }
I20260619 16:43:32.830710 3494 raft_consensus.cc:388] T 00000000000000000000000000000000 P 01db4b7096f343038b0995c1a919b60d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:43:32.830832 3494 raft_consensus.cc:482] T 00000000000000000000000000000000 P 01db4b7096f343038b0995c1a919b60d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:43:32.830974 3494 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 01db4b7096f343038b0995c1a919b60d [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:43:32.832988 3494 raft_consensus.cc:504] T 00000000000000000000000000000000 P 01db4b7096f343038b0995c1a919b60d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01db4b7096f343038b0995c1a919b60d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41885 } }
I20260619 16:43:32.833416 3494 leader_election.cc:302] T 00000000000000000000000000000000 P 01db4b7096f343038b0995c1a919b60d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 01db4b7096f343038b0995c1a919b60d; no voters:
I20260619 16:43:32.833947 3494 leader_election.cc:288] T 00000000000000000000000000000000 P 01db4b7096f343038b0995c1a919b60d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 16:43:32.834034 3499 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 01db4b7096f343038b0995c1a919b60d [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:43:32.834611 3499 raft_consensus.cc:686] T 00000000000000000000000000000000 P 01db4b7096f343038b0995c1a919b60d [term 1 LEADER]: Becoming Leader. State: Replica: 01db4b7096f343038b0995c1a919b60d, State: Running, Role: LEADER
I20260619 16:43:32.835201 3499 consensus_queue.cc:237] T 00000000000000000000000000000000 P 01db4b7096f343038b0995c1a919b60d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01db4b7096f343038b0995c1a919b60d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41885 } }
I20260619 16:43:32.835965 3494 sys_catalog.cc:564] T 00000000000000000000000000000000 P 01db4b7096f343038b0995c1a919b60d [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:43:32.837671 3501 sys_catalog.cc:455] T 00000000000000000000000000000000 P 01db4b7096f343038b0995c1a919b60d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 01db4b7096f343038b0995c1a919b60d. Latest consensus state: current_term: 1 leader_uuid: "01db4b7096f343038b0995c1a919b60d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01db4b7096f343038b0995c1a919b60d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41885 } } }
I20260619 16:43:32.837780 3500 sys_catalog.cc:455] T 00000000000000000000000000000000 P 01db4b7096f343038b0995c1a919b60d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "01db4b7096f343038b0995c1a919b60d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01db4b7096f343038b0995c1a919b60d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41885 } } }
I20260619 16:43:32.838058 3500 sys_catalog.cc:458] T 00000000000000000000000000000000 P 01db4b7096f343038b0995c1a919b60d [sys.catalog]: This master's current role is: LEADER
I20260619 16:43:32.838066 3501 sys_catalog.cc:458] T 00000000000000000000000000000000 P 01db4b7096f343038b0995c1a919b60d [sys.catalog]: This master's current role is: LEADER
I20260619 16:43:32.839411 3504 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:43:32.844117 3504 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:43:32.852394 3504 catalog_manager.cc:1347] Generated new cluster ID: 930f51bfb4584b4da281e4862529f488
I20260619 16:43:32.852491 3504 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:43:32.864964 3504 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 16:43:32.866358 3504 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:43:32.878775 3504 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 01db4b7096f343038b0995c1a919b60d: Generated new TSK 0
I20260619 16:43:32.879711 3504 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887385903558-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887385903558-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260619 16:43:32.929133 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu
/tmp/dist-test-taskEs2EH8/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887385903558-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887385903558-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887385903558-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887385903558-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41885
--keytab_file=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887385903558-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41687
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887385903558-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887385903558-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887385903558-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:43:33.237553 3520 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:43:33.237859 3520 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:43:33.238004 3520 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:43:33.245045 3520 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:43:33.245158 3520 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:43:33.245308 3520 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260619 16:43:33.254702 3520 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41687
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887385903558-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887385903558-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887385903558-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887385903558-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41885
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887385903558-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:43:33.256044 3520 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:43:33.257308 3520 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:43:33.269505 3525 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:43:33.269567 3528 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:43:33.269779 3526 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:43:33.270543 3520 server_base.cc:1034] running on GCE node
I20260619 16:43:33.271087 3520 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:43:33.271942 3520 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:43:33.273129 3520 hybrid_clock.cc:648] HybridClock initialized: now 1781887413273085 us; error 38 us; skew 500 ppm
I20260619 16:43:33.273413 3520 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:43:33 dist-test-slave-wvz7 krb5kdc[3414](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887413, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:43:33.276305 3520 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260619 16:43:33.277558 3520 webserver.cc:460] Webserver started at http://127.0.97.1:42787/ using document root <none> and password file <none>
I20260619 16:43:33.278251 3520 fs_manager.cc:359] Metadata directory not provided
I20260619 16:43:33.278353 3520 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:43:33.278620 3520 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:43:33.281265 3520 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887385903558-388-0/minicluster-data/ts-0/data/instance:
uuid: "faab8e07d98043c598929d5f0ae56472"
format_stamp: "Formatted at 2026-06-19 16:43:33 on dist-test-slave-wvz7"
server_key: "a6471288020729354ffe1711f6aa82fa"
server_key_iv: "2d48f8485f39ed9661682213ffec9062"
server_key_version: "encryptionkey@0"
I20260619 16:43:33.282040 3520 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887385903558-388-0/minicluster-data/ts-0/wal/instance:
uuid: "faab8e07d98043c598929d5f0ae56472"
format_stamp: "Formatted at 2026-06-19 16:43:33 on dist-test-slave-wvz7"
server_key: "a6471288020729354ffe1711f6aa82fa"
server_key_iv: "2d48f8485f39ed9661682213ffec9062"
server_key_version: "encryptionkey@0"
I20260619 16:43:33.286825 3520 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260619 16:43:33.290055 3535 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:43:33.291373 3520 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260619 16:43:33.291527 3520 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887385903558-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887385903558-388-0/minicluster-data/ts-0/wal
uuid: "faab8e07d98043c598929d5f0ae56472"
format_stamp: "Formatted at 2026-06-19 16:43:33 on dist-test-slave-wvz7"
server_key: "a6471288020729354ffe1711f6aa82fa"
server_key_iv: "2d48f8485f39ed9661682213ffec9062"
server_key_version: "encryptionkey@0"
I20260619 16:43:33.291697 3520 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887385903558-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887385903558-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887385903558-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:43:33.306756 3520 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:43:33.310406 3520 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:43:33.310660 3520 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:43:33.311648 3520 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:43:33.313087 3520 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:43:33.313156 3520 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:43:33.313277 3520 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:43:33.313323 3520 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:43:33.343585 3520 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:39613
I20260619 16:43:33.343628 3647 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:39613 every 8 connection(s)
I20260619 16:43:33.344806 3520 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887385903558-388-0/minicluster-data/ts-0/data/info.pb
I20260619 16:43:33.345678 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu as pid 3520
I20260619 16:43:33.345899 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887385903558-388-0/minicluster-data/ts-0/wal/instance
I20260619 16:43:33.346674 388 external_mini_cluster.cc:1428] Setting key 8c6d38a2282d031f65d43d3bdc80a8d0
Jun 19 16:43:33 dist-test-slave-wvz7 krb5kdc[3414](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887413, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260619 16:43:33.365334 3650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:33.347525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:42469 (local address 127.0.97.62:41885)
0619 16:43:33.348651 (+ 1126us) server_negotiation.cc:206] Beginning negotiation
0619 16:43:33.348657 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:43:33.349201 (+ 544us) server_negotiation.cc:407] Connection header received
0619 16:43:33.351548 (+ 2347us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:43:33.351552 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:43:33.351616 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:43:33.351704 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:43:33.353486 (+ 1782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:33.354258 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:33.355740 (+ 1482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:33.356291 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:33.356415 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:43:33.359687 (+ 3272us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:43:33.359692 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:43:33.359695 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:43:33.359724 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:43:33.362127 (+ 2403us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:43:33.362974 (+ 847us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:43:33.362980 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:43:33.362984 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:43:33.363187 (+ 203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:43:33.363653 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:43:33.363657 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:43:33.363659 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:43:33.363910 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:43:33.364023 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0619 16:43:33.364793 (+ 770us) server_negotiation.cc:299] Negotiation successful
0619 16:43:33.364957 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1010,"thread_start_us":910,"threads_started":1}
I20260619 16:43:33.366183 3651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:33.348027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41885 (local address 127.0.97.1:42469)
0619 16:43:33.349027 (+ 1000us) negotiation.cc:107] Waiting for socket to connect
0619 16:43:33.349079 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0619 16:43:33.351270 (+ 2191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:43:33.351948 (+ 678us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:43:33.351959 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:43:33.352607 (+ 648us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:43:33.353321 (+ 714us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:43:33.353342 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:43:33.354405 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:43:33.354409 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:43:33.355606 (+ 1197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:43:33.355617 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:43:33.356476 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:43:33.356480 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:43:33.356680 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:43:33.357261 (+ 581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:43:33.357286 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:43:33.359492 (+ 2206us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:43:33.362251 (+ 2759us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:43:33.362260 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:43:33.362275 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:43:33.362850 (+ 575us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:43:33.363314 (+ 464us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:43:33.363320 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:43:33.363324 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:43:33.363553 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:43:33.364028 (+ 475us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:43:33.364036 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:43:33.364427 (+ 391us) client_negotiation.cc:769] Sending connection context
0619 16:43:33.364720 (+ 293us) client_negotiation.cc:240] Negotiation successful
0619 16:43:33.365076 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":729,"thread_start_us":555,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260619 16:43:33.367686 3648 heartbeater.cc:347] Connected to a master server at 127.0.97.62:41885
I20260619 16:43:33.367998 3648 heartbeater.cc:464] Registering TS with master...
I20260619 16:43:33.368669 3648 heartbeater.cc:511] Master 127.0.97.62:41885 requested a full tablet report, sending...
I20260619 16:43:33.370587 3459 ts_manager.cc:194] Registered new tserver with Master: faab8e07d98043c598929d5f0ae56472 (127.0.97.1:39613)
I20260619 16:43:33.372442 3459 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:42469
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887385903558-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887385903558-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260619 16:43:33.385563 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu
/tmp/dist-test-taskEs2EH8/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887385903558-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887385903558-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887385903558-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887385903558-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41885
--keytab_file=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887385903558-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41687
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887385903558-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887385903558-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887385903558-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:43:33.682893 3654 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:43:33.683161 3654 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:43:33.683255 3654 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:43:33.690209 3654 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:43:33.690310 3654 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:43:33.690423 3654 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260619 16:43:33.699786 3654 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41687
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887385903558-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887385903558-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887385903558-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887385903558-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41885
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887385903558-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:43:33.700961 3654 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:43:33.702211 3654 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:43:33.714605 3662 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:43:33.714676 3659 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:43:33.714710 3660 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:43:33.715251 3654 server_base.cc:1034] running on GCE node
I20260619 16:43:33.715808 3654 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:43:33.716744 3654 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:43:33.717952 3654 hybrid_clock.cc:648] HybridClock initialized: now 1781887413717893 us; error 38 us; skew 500 ppm
I20260619 16:43:33.718242 3654 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:43:33 dist-test-slave-wvz7 krb5kdc[3414](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887413, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:43:33.721158 3654 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260619 16:43:33.722537 3654 webserver.cc:460] Webserver started at http://127.0.97.2:46815/ using document root <none> and password file <none>
I20260619 16:43:33.723181 3654 fs_manager.cc:359] Metadata directory not provided
I20260619 16:43:33.723285 3654 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:43:33.723549 3654 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:43:33.726375 3654 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887385903558-388-0/minicluster-data/ts-1/data/instance:
uuid: "8a0f7b31dfcc42d4b60bb6a41f7fd4c8"
format_stamp: "Formatted at 2026-06-19 16:43:33 on dist-test-slave-wvz7"
server_key: "fc548ee2f7677d6aa0df3e09769cfa4a"
server_key_iv: "7f582f09e10f335e8ce569fe18a05fc2"
server_key_version: "encryptionkey@0"
I20260619 16:43:33.727099 3654 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887385903558-388-0/minicluster-data/ts-1/wal/instance:
uuid: "8a0f7b31dfcc42d4b60bb6a41f7fd4c8"
format_stamp: "Formatted at 2026-06-19 16:43:33 on dist-test-slave-wvz7"
server_key: "fc548ee2f7677d6aa0df3e09769cfa4a"
server_key_iv: "7f582f09e10f335e8ce569fe18a05fc2"
server_key_version: "encryptionkey@0"
I20260619 16:43:33.731846 3654 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 16:43:33.734961 3669 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:43:33.736228 3654 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 16:43:33.736362 3654 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887385903558-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887385903558-388-0/minicluster-data/ts-1/wal
uuid: "8a0f7b31dfcc42d4b60bb6a41f7fd4c8"
format_stamp: "Formatted at 2026-06-19 16:43:33 on dist-test-slave-wvz7"
server_key: "fc548ee2f7677d6aa0df3e09769cfa4a"
server_key_iv: "7f582f09e10f335e8ce569fe18a05fc2"
server_key_version: "encryptionkey@0"
I20260619 16:43:33.736523 3654 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887385903558-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887385903558-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887385903558-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:43:33.750319 3654 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:43:33.754019 3654 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:43:33.754276 3654 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:43:33.755191 3654 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:43:33.756460 3654 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:43:33.756532 3654 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:43:33.756645 3654 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:43:33.756701 3654 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:43:33.785477 3654 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:34565
I20260619 16:43:33.785542 3781 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:34565 every 8 connection(s)
I20260619 16:43:33.786691 3654 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887385903558-388-0/minicluster-data/ts-1/data/info.pb
I20260619 16:43:33.790594 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu as pid 3654
I20260619 16:43:33.790766 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887385903558-388-0/minicluster-data/ts-1/wal/instance
I20260619 16:43:33.791289 388 external_mini_cluster.cc:1428] Setting key d67ea4c8dd4d57408af514235cb6d060
Jun 19 16:43:33 dist-test-slave-wvz7 krb5kdc[3414](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887413, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260619 16:43:33.809798 3650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:33.789138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:33417 (local address 127.0.97.62:41885)
0619 16:43:33.789348 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0619 16:43:33.789354 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:43:33.790503 (+ 1149us) server_negotiation.cc:407] Connection header received
0619 16:43:33.792669 (+ 2166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:43:33.792675 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:43:33.792758 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:43:33.792877 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:43:33.794912 (+ 2035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:33.796066 (+ 1154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:33.798138 (+ 2072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:33.798780 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:33.798904 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:43:33.803100 (+ 4196us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:43:33.803107 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:43:33.803112 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:43:33.803149 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:43:33.806504 (+ 3355us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:43:33.807357 (+ 853us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:43:33.807363 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:43:33.807367 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:43:33.807525 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:43:33.807995 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:43:33.808000 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:43:33.808004 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:43:33.808327 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:43:33.808476 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0619 16:43:33.809333 (+ 857us) server_negotiation.cc:299] Negotiation successful
0619 16:43:33.809520 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260619 16:43:33.810909 3784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:33.789595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41885 (local address 127.0.97.2:33417)
0619 16:43:33.790325 (+ 730us) negotiation.cc:107] Waiting for socket to connect
0619 16:43:33.790380 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0619 16:43:33.792395 (+ 2015us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:43:33.793110 (+ 715us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:43:33.793118 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:43:33.793851 (+ 733us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:43:33.794711 (+ 860us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:43:33.794738 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:43:33.796218 (+ 1480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:43:33.796223 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:43:33.798006 (+ 1783us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:43:33.798019 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:43:33.799018 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:43:33.799025 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:43:33.799304 (+ 279us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:43:33.800056 (+ 752us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:43:33.800085 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:43:33.802846 (+ 2761us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:43:33.806652 (+ 3806us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:43:33.806661 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:43:33.806676 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:43:33.807224 (+ 548us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:43:33.807638 (+ 414us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:43:33.807643 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:43:33.807647 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:43:33.807867 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:43:33.808468 (+ 601us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:43:33.808475 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:43:33.808865 (+ 390us) client_negotiation.cc:769] Sending connection context
0619 16:43:33.809119 (+ 254us) client_negotiation.cc:240] Negotiation successful
0619 16:43:33.809633 (+ 514us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":475,"thread_start_us":277,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
I20260619 16:43:33.812222 3782 heartbeater.cc:347] Connected to a master server at 127.0.97.62:41885
I20260619 16:43:33.812556 3782 heartbeater.cc:464] Registering TS with master...
I20260619 16:43:33.813407 3782 heartbeater.cc:511] Master 127.0.97.62:41885 requested a full tablet report, sending...
I20260619 16:43:33.814924 3459 ts_manager.cc:194] Registered new tserver with Master: 8a0f7b31dfcc42d4b60bb6a41f7fd4c8 (127.0.97.2:34565)
I20260619 16:43:33.815714 3459 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:33417
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887385903558-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887385903558-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260619 16:43:33.830798 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu
/tmp/dist-test-taskEs2EH8/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887385903558-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887385903558-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887385903558-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887385903558-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41885
--keytab_file=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887385903558-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41687
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887385903558-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887385903558-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887385903558-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:43:34.144825 3787 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:43:34.145090 3787 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:43:34.145217 3787 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:43:34.152123 3787 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:43:34.152235 3787 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:43:34.152376 3787 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260619 16:43:34.161418 3787 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41687
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887385903558-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887385903558-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887385903558-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887385903558-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41885
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887385903558-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:43:34.162586 3787 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:43:34.163770 3787 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:43:34.175513 3792 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:43:34.175585 3793 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:43:34.175555 3795 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:43:34.176932 3787 server_base.cc:1034] running on GCE node
I20260619 16:43:34.177518 3787 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:43:34.178438 3787 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:43:34.179651 3787 hybrid_clock.cc:648] HybridClock initialized: now 1781887414179607 us; error 45 us; skew 500 ppm
I20260619 16:43:34.179937 3787 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:43:34 dist-test-slave-wvz7 krb5kdc[3414](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887414, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:43:34.183043 3787 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260619 16:43:34.184490 3787 webserver.cc:460] Webserver started at http://127.0.97.3:46465/ using document root <none> and password file <none>
I20260619 16:43:34.185146 3787 fs_manager.cc:359] Metadata directory not provided
I20260619 16:43:34.185256 3787 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:43:34.185531 3787 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:43:34.188303 3787 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887385903558-388-0/minicluster-data/ts-2/data/instance:
uuid: "d6cb7387192047ed8e4fcb7fce64a9a7"
format_stamp: "Formatted at 2026-06-19 16:43:34 on dist-test-slave-wvz7"
server_key: "7f406f0d1a6f7212dc315fca8ec9a58a"
server_key_iv: "eca16254e2a8d57518e9691c5778eddb"
server_key_version: "encryptionkey@0"
I20260619 16:43:34.189009 3787 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887385903558-388-0/minicluster-data/ts-2/wal/instance:
uuid: "d6cb7387192047ed8e4fcb7fce64a9a7"
format_stamp: "Formatted at 2026-06-19 16:43:34 on dist-test-slave-wvz7"
server_key: "7f406f0d1a6f7212dc315fca8ec9a58a"
server_key_iv: "eca16254e2a8d57518e9691c5778eddb"
server_key_version: "encryptionkey@0"
I20260619 16:43:34.193773 3787 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260619 16:43:34.197039 3802 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:43:34.198457 3787 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 16:43:34.198601 3787 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887385903558-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887385903558-388-0/minicluster-data/ts-2/wal
uuid: "d6cb7387192047ed8e4fcb7fce64a9a7"
format_stamp: "Formatted at 2026-06-19 16:43:34 on dist-test-slave-wvz7"
server_key: "7f406f0d1a6f7212dc315fca8ec9a58a"
server_key_iv: "eca16254e2a8d57518e9691c5778eddb"
server_key_version: "encryptionkey@0"
I20260619 16:43:34.198770 3787 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887385903558-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887385903558-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887385903558-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:43:34.215863 3787 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:43:34.219719 3787 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:43:34.219965 3787 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:43:34.220808 3787 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:43:34.222038 3787 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:43:34.222114 3787 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:43:34.222225 3787 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:43:34.222276 3787 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:43:34.250556 3787 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:34333
I20260619 16:43:34.250618 3914 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:34333 every 8 connection(s)
I20260619 16:43:34.251713 3787 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887385903558-388-0/minicluster-data/ts-2/data/info.pb
I20260619 16:43:34.256613 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu as pid 3787
I20260619 16:43:34.256789 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887385903558-388-0/minicluster-data/ts-2/wal/instance
I20260619 16:43:34.257318 388 external_mini_cluster.cc:1428] Setting key 556a452730455838f61b75e0a4e38fa0
Jun 19 16:43:34 dist-test-slave-wvz7 krb5kdc[3414](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887414, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 16:43:34.271449 3650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:34.254227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:49383 (local address 127.0.97.62:41885)
0619 16:43:34.254427 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0619 16:43:34.254432 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:43:34.255745 (+ 1313us) server_negotiation.cc:407] Connection header received
0619 16:43:34.258229 (+ 2484us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:43:34.258243 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:43:34.258324 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:43:34.258424 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:43:34.260025 (+ 1601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:34.260742 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:34.262279 (+ 1537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:34.262673 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:34.262780 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:43:34.265970 (+ 3190us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:43:34.265975 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:43:34.265978 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:43:34.266008 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:43:34.268479 (+ 2471us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:43:34.269190 (+ 711us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:43:34.269196 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:43:34.269198 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:43:34.269317 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:43:34.269759 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:43:34.269765 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:43:34.269769 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:43:34.270073 (+ 304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:43:34.270213 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0619 16:43:34.270949 (+ 736us) server_negotiation.cc:299] Negotiation successful
0619 16:43:34.271155 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260619 16:43:34.272555 3917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:34.254729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41885 (local address 127.0.97.3:49383)
0619 16:43:34.255570 (+ 841us) negotiation.cc:107] Waiting for socket to connect
0619 16:43:34.255616 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0619 16:43:34.257975 (+ 2359us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:43:34.258716 (+ 741us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:43:34.258724 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:43:34.259303 (+ 579us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:43:34.259865 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:43:34.259881 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:43:34.260886 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:43:34.260890 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:43:34.262154 (+ 1264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:43:34.262165 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:43:34.262874 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:43:34.262881 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:43:34.263107 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:43:34.263681 (+ 574us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:43:34.263703 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:43:34.265763 (+ 2060us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:43:34.268618 (+ 2855us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:43:34.268628 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:43:34.268644 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:43:34.269060 (+ 416us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:43:34.269423 (+ 363us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:43:34.269429 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:43:34.269433 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:43:34.269619 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:43:34.270199 (+ 580us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:43:34.270207 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:43:34.270556 (+ 349us) client_negotiation.cc:769] Sending connection context
0619 16:43:34.270826 (+ 270us) client_negotiation.cc:240] Negotiation successful
0619 16:43:34.271289 (+ 463us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":534,"thread_start_us":334,"threads_started":1}
I20260619 16:43:34.273701 3915 heartbeater.cc:347] Connected to a master server at 127.0.97.62:41885
I20260619 16:43:34.274040 3915 heartbeater.cc:464] Registering TS with master...
I20260619 16:43:34.274679 3915 heartbeater.cc:511] Master 127.0.97.62:41885 requested a full tablet report, sending...
I20260619 16:43:34.275883 3459 ts_manager.cc:194] Registered new tserver with Master: d6cb7387192047ed8e4fcb7fce64a9a7 (127.0.97.3:34333)
I20260619 16:43:34.276613 3459 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:49383
I20260619 16:43:34.284318 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 19 16:43:34 dist-test-slave-wvz7 krb5kdc[3414](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887414, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Jun 19 16:43:34 dist-test-slave-wvz7 krb5kdc[3414](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887414, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 16:43:34.325819 3650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:34.313301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54848 (local address 127.0.97.62:41885)
0619 16:43:34.313477 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0619 16:43:34.313483 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:43:34.313925 (+ 442us) server_negotiation.cc:407] Connection header received
0619 16:43:34.314106 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:43:34.314110 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:43:34.314169 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:43:34.314251 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:43:34.315495 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:34.316192 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:34.317352 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:34.317776 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:34.317885 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:43:34.321235 (+ 3350us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:43:34.321240 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:43:34.321243 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:43:34.321275 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:43:34.323455 (+ 2180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:43:34.324029 (+ 574us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:43:34.324033 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:43:34.324036 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:43:34.324150 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:43:34.324512 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:43:34.324517 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:43:34.324521 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:43:34.324806 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:43:34.325024 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0619 16:43:34.325310 (+ 286us) server_negotiation.cc:299] Negotiation successful
0619 16:43:34.325505 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
W20260619 16:43:34.326320 3459 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:54848
I20260619 16:43:34.327644 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu pid 3520
Jun 19 16:43:34 dist-test-slave-wvz7 krb5kdc[3414](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887414, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260619 16:43:34.342496 3926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:34.328943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42806 (local address 127.0.97.1:39613)
0619 16:43:34.329551 (+ 608us) server_negotiation.cc:206] Beginning negotiation
0619 16:43:34.329560 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 16:43:34.329589 (+ 29us) server_negotiation.cc:407] Connection header received
0619 16:43:34.329651 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:43:34.329659 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:43:34.329952 (+ 293us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:43:34.330120 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:43:34.331320 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:34.332401 (+ 1081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:34.333532 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:34.334175 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:34.334282 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:43:34.337695 (+ 3413us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:43:34.337705 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:43:34.337718 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:43:34.337776 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:43:34.340386 (+ 2610us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:43:34.340936 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:43:34.340943 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:43:34.340945 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:43:34.341059 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:43:34.341317 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:43:34.341320 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:43:34.341322 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:43:34.341611 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:43:34.341868 (+ 257us) server_negotiation.cc:1035] Waiting for connection context
0619 16:43:34.342044 (+ 176us) server_negotiation.cc:299] Negotiation successful
0619 16:43:34.342219 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":463,"thread_start_us":340,"threads_started":1}
W20260619 16:43:34.343777 3643 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:42806
W20260619 16:43:34.344997 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu pid 3520: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 16:43:34.345180 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu with pid 3520
I20260619 16:43:34.356696 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu pid 3654
Jun 19 16:43:34 dist-test-slave-wvz7 krb5kdc[3414](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887414, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260619 16:43:34.371675 3927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:34.357515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54472 (local address 127.0.97.2:34565)
0619 16:43:34.358071 (+ 556us) server_negotiation.cc:206] Beginning negotiation
0619 16:43:34.358078 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:43:34.358103 (+ 25us) server_negotiation.cc:407] Connection header received
0619 16:43:34.358159 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:43:34.358165 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:43:34.358371 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:43:34.358532 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:43:34.359738 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:34.360751 (+ 1013us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:34.361998 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:34.362550 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:34.362671 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:43:34.366449 (+ 3778us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:43:34.366459 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:43:34.366473 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:43:34.366505 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:43:34.369308 (+ 2803us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:43:34.369961 (+ 653us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:43:34.369970 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:43:34.369973 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:43:34.370111 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:43:34.370442 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:43:34.370447 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:43:34.370449 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:43:34.370806 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:43:34.371041 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0619 16:43:34.371228 (+ 187us) server_negotiation.cc:299] Negotiation successful
0619 16:43:34.371408 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":427,"thread_start_us":288,"threads_started":1}
W20260619 16:43:34.372797 3777 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:54472
W20260619 16:43:34.373898 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu pid 3654: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 16:43:34.374027 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu with pid 3654
I20260619 16:43:34.385566 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu pid 3787
Jun 19 16:43:34 dist-test-slave-wvz7 krb5kdc[3414](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887414, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260619 16:43:34.399830 3928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:34.386454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49426 (local address 127.0.97.3:34333)
0619 16:43:34.387029 (+ 575us) server_negotiation.cc:206] Beginning negotiation
0619 16:43:34.387035 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:43:34.387061 (+ 26us) server_negotiation.cc:407] Connection header received
0619 16:43:34.387119 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:43:34.387124 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:43:34.387312 (+ 188us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:43:34.387444 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:43:34.388571 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:34.389537 (+ 966us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:34.390773 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:34.391296 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:34.391398 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:43:34.395184 (+ 3786us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:43:34.395194 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:43:34.395207 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:43:34.395238 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:43:34.397620 (+ 2382us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:43:34.398263 (+ 643us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:43:34.398270 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:43:34.398272 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:43:34.398384 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:43:34.398657 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:43:34.398660 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:43:34.398662 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:43:34.398944 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:43:34.399153 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0619 16:43:34.399382 (+ 229us) server_negotiation.cc:299] Negotiation successful
0619 16:43:34.399560 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":449,"thread_start_us":306,"threads_started":1}
W20260619 16:43:34.400949 3910 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:49426
W20260619 16:43:34.401991 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu pid 3787: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 16:43:34.402154 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu with pid 3787
I20260619 16:43:34.413324 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu pid 3427
I20260619 16:43:34.414322 3489 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:43:34.513330 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu with pid 3427
2026-06-19T16:43:34Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (3323 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887385903558-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 16:43:34 dist-test-slave-wvz7 krb5kdc[3932](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 16:43:34 dist-test-slave-wvz7 krb5kdc[3932](info): set up 2 sockets
Jun 19 16:43:34 dist-test-slave-wvz7 krb5kdc[3932](info): commencing operation
krb5kdc: starting...
W20260619 16:43:35.501973 388 mini_kdc.cc:121] Time spent starting KDC: real 0.953s user 0.000s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887385903558-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887385903558-388-0/krb5kdc/test-user.keytab.
Jun 19 16:43:35 dist-test-slave-wvz7 krb5kdc[3932](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887415, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T16:43:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T16:43:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887385903558-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887385903558-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260619 16:43:35.673528 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu
/tmp/dist-test-taskEs2EH8/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887385903558-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887385903558-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887385903558-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887385903558-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:43381
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887385903558-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40819
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:43381
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887385903558-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887385903558-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887385903558-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:43:35.973358 3945 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:43:35.973644 3945 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:43:35.973803 3945 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:43:35.980881 3945 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:43:35.980991 3945 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:43:35.981072 3945 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:43:35.981148 3945 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:43:35.981206 3945 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:43:35.990864 3945 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40819
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887385903558-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887385903558-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:43381
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887385903558-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:43381
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887385903558-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887385903558-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:43:35.992103 3945 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:43:35.993348 3945 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:43:36.004150 3950 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:43:36.004181 3951 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:43:36.004117 3953 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:43:36.004777 3945 server_base.cc:1034] running on GCE node
I20260619 16:43:36.005487 3945 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:43:36.006800 3945 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:43:36.008009 3945 hybrid_clock.cc:648] HybridClock initialized: now 1781887416007958 us; error 38 us; skew 500 ppm
I20260619 16:43:36.008324 3945 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:43:36 dist-test-slave-wvz7 krb5kdc[3932](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887416, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:43:36.014392 3945 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260619 16:43:36.015791 3945 webserver.cc:460] Webserver started at http://127.0.97.62:42815/ using document root <none> and password file <none>
I20260619 16:43:36.016443 3945 fs_manager.cc:359] Metadata directory not provided
I20260619 16:43:36.016556 3945 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:43:36.016837 3945 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:43:36.019747 3945 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887385903558-388-0/minicluster-data/master-0/data/instance:
uuid: "6085fade47ae409b8ac2e912156340b7"
format_stamp: "Formatted at 2026-06-19 16:43:36 on dist-test-slave-wvz7"
server_key: "cb14e5b66aec393d04365b93770a3ee3"
server_key_iv: "8da2766ea4b0cbe61459f8484c69cdbf"
server_key_version: "encryptionkey@0"
I20260619 16:43:36.020468 3945 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887385903558-388-0/minicluster-data/master-0/wal/instance:
uuid: "6085fade47ae409b8ac2e912156340b7"
format_stamp: "Formatted at 2026-06-19 16:43:36 on dist-test-slave-wvz7"
server_key: "cb14e5b66aec393d04365b93770a3ee3"
server_key_iv: "8da2766ea4b0cbe61459f8484c69cdbf"
server_key_version: "encryptionkey@0"
I20260619 16:43:36.025280 3945 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260619 16:43:36.028700 3960 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:43:36.030043 3945 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 16:43:36.030192 3945 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887385903558-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887385903558-388-0/minicluster-data/master-0/wal
uuid: "6085fade47ae409b8ac2e912156340b7"
format_stamp: "Formatted at 2026-06-19 16:43:36 on dist-test-slave-wvz7"
server_key: "cb14e5b66aec393d04365b93770a3ee3"
server_key_iv: "8da2766ea4b0cbe61459f8484c69cdbf"
server_key_version: "encryptionkey@0"
I20260619 16:43:36.030368 3945 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887385903558-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887385903558-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887385903558-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:43:36.057821 3945 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:43:36.061875 3945 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:43:36.062124 3945 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:43:36.078001 3945 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:43381
I20260619 16:43:36.078032 4011 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:43381 every 8 connection(s)
I20260619 16:43:36.079286 3945 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887385903558-388-0/minicluster-data/master-0/data/info.pb
I20260619 16:43:36.082747 4012 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:43:36.089505 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu as pid 3945
I20260619 16:43:36.089699 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887385903558-388-0/minicluster-data/master-0/wal/instance
I20260619 16:43:36.090278 388 external_mini_cluster.cc:1428] Setting key e13ecf9c40c613172e1c71b95d2014c9
I20260619 16:43:36.092410 4012 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6085fade47ae409b8ac2e912156340b7: Bootstrap starting.
I20260619 16:43:36.095510 4012 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6085fade47ae409b8ac2e912156340b7: Neither blocks nor log segments found. Creating new log.
I20260619 16:43:36.096738 4012 log.cc:824] T 00000000000000000000000000000000 P 6085fade47ae409b8ac2e912156340b7: Log is configured to *not* fsync() on all Append() calls
I20260619 16:43:36.099576 4012 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6085fade47ae409b8ac2e912156340b7: No bootstrap required, opened a new log
Jun 19 16:43:36 dist-test-slave-wvz7 krb5kdc[3932](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887415, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 16:43:36.107896 4012 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6085fade47ae409b8ac2e912156340b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6085fade47ae409b8ac2e912156340b7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43381 } }
I20260619 16:43:36.108319 4012 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6085fade47ae409b8ac2e912156340b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:43:36.108420 4012 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6085fade47ae409b8ac2e912156340b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6085fade47ae409b8ac2e912156340b7, State: Initialized, Role: FOLLOWER
I20260619 16:43:36.109066 4012 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6085fade47ae409b8ac2e912156340b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6085fade47ae409b8ac2e912156340b7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43381 } }
I20260619 16:43:36.109345 4012 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6085fade47ae409b8ac2e912156340b7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:43:36.109480 4012 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6085fade47ae409b8ac2e912156340b7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:43:36.109635 4012 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6085fade47ae409b8ac2e912156340b7 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:43:36.111789 4015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:36.092063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33238 (local address 127.0.97.62:43381)
0619 16:43:36.092748 (+ 685us) server_negotiation.cc:206] Beginning negotiation
0619 16:43:36.092760 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 16:43:36.092800 (+ 40us) server_negotiation.cc:407] Connection header received
0619 16:43:36.094490 (+ 1690us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:43:36.094523 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:43:36.094870 (+ 347us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:43:36.095162 (+ 292us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:43:36.096897 (+ 1735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:36.098232 (+ 1335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:36.099894 (+ 1662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:36.100542 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:36.100665 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:43:36.105056 (+ 4391us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:43:36.105066 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:43:36.105081 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:43:36.105115 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:43:36.107887 (+ 2772us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:43:36.108638 (+ 751us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:43:36.108647 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:43:36.108654 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:43:36.108856 (+ 202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:43:36.109225 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:43:36.109228 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:43:36.109230 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:43:36.109836 (+ 606us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:43:36.110048 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0619 16:43:36.110453 (+ 405us) server_negotiation.cc:299] Negotiation successful
0619 16:43:36.110812 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":509,"thread_start_us":321,"threads_started":1}
I20260619 16:43:36.112186 4012 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6085fade47ae409b8ac2e912156340b7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6085fade47ae409b8ac2e912156340b7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43381 } }
I20260619 16:43:36.112635 4012 leader_election.cc:302] T 00000000000000000000000000000000 P 6085fade47ae409b8ac2e912156340b7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6085fade47ae409b8ac2e912156340b7; no voters:
I20260619 16:43:36.113178 4012 leader_election.cc:288] T 00000000000000000000000000000000 P 6085fade47ae409b8ac2e912156340b7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 16:43:36.113298 4017 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6085fade47ae409b8ac2e912156340b7 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:43:36.114991 4017 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6085fade47ae409b8ac2e912156340b7 [term 1 LEADER]: Becoming Leader. State: Replica: 6085fade47ae409b8ac2e912156340b7, State: Running, Role: LEADER
I20260619 16:43:36.115269 4012 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6085fade47ae409b8ac2e912156340b7 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:43:36.115566 4017 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6085fade47ae409b8ac2e912156340b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6085fade47ae409b8ac2e912156340b7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43381 } }
I20260619 16:43:36.118224 4019 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6085fade47ae409b8ac2e912156340b7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6085fade47ae409b8ac2e912156340b7. Latest consensus state: current_term: 1 leader_uuid: "6085fade47ae409b8ac2e912156340b7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6085fade47ae409b8ac2e912156340b7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43381 } } }
I20260619 16:43:36.118567 4019 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6085fade47ae409b8ac2e912156340b7 [sys.catalog]: This master's current role is: LEADER
I20260619 16:43:36.119067 4018 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6085fade47ae409b8ac2e912156340b7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6085fade47ae409b8ac2e912156340b7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6085fade47ae409b8ac2e912156340b7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43381 } } }
I20260619 16:43:36.119346 4018 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6085fade47ae409b8ac2e912156340b7 [sys.catalog]: This master's current role is: LEADER
W20260619 16:43:36.126469 4033 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 6085fade47ae409b8ac2e912156340b7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 16:43:36.126592 4033 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 16:43:36.126713 4025 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:43:36.128147 4025 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:43:36.135272 4025 catalog_manager.cc:1347] Generated new cluster ID: 91eebbff56d94b77b64526512035ed17
I20260619 16:43:36.135375 4025 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:43:36.151603 4025 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 16:43:36.152942 4025 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:43:36.162144 4025 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6085fade47ae409b8ac2e912156340b7: Generated new TSK 0
I20260619 16:43:36.162930 4025 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887385903558-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887385903558-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260619 16:43:36.209127 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu
/tmp/dist-test-taskEs2EH8/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887385903558-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887385903558-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887385903558-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887385903558-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43381
--keytab_file=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887385903558-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40819
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887385903558-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887385903558-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887385903558-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:43:36.515312 4038 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:43:36.515602 4038 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:43:36.515710 4038 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:43:36.523108 4038 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:43:36.523223 4038 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:43:36.523350 4038 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260619 16:43:36.533046 4038 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40819
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887385903558-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887385903558-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887385903558-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887385903558-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43381
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887385903558-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:43:36.534437 4038 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:43:36.535722 4038 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:43:36.548022 4046 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:43:36.548077 4044 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:43:36.548045 4043 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:43:36.549283 4038 server_base.cc:1034] running on GCE node
I20260619 16:43:36.549888 4038 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:43:36.550807 4038 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:43:36.552028 4038 hybrid_clock.cc:648] HybridClock initialized: now 1781887416551971 us; error 47 us; skew 500 ppm
I20260619 16:43:36.552328 4038 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:43:36 dist-test-slave-wvz7 krb5kdc[3932](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887416, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:43:36.555307 4038 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260619 16:43:36.556605 4038 webserver.cc:460] Webserver started at http://127.0.97.1:44003/ using document root <none> and password file <none>
I20260619 16:43:36.557260 4038 fs_manager.cc:359] Metadata directory not provided
I20260619 16:43:36.557354 4038 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:43:36.557621 4038 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:43:36.560390 4038 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887385903558-388-0/minicluster-data/ts-0/data/instance:
uuid: "3aae94a9a9da4e7abb6b2af5a691dfcf"
format_stamp: "Formatted at 2026-06-19 16:43:36 on dist-test-slave-wvz7"
server_key: "bd800c079077b20bdacc37e3fcf13957"
server_key_iv: "2614d5d88e25c500a6fc9872d1399677"
server_key_version: "encryptionkey@0"
I20260619 16:43:36.561117 4038 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887385903558-388-0/minicluster-data/ts-0/wal/instance:
uuid: "3aae94a9a9da4e7abb6b2af5a691dfcf"
format_stamp: "Formatted at 2026-06-19 16:43:36 on dist-test-slave-wvz7"
server_key: "bd800c079077b20bdacc37e3fcf13957"
server_key_iv: "2614d5d88e25c500a6fc9872d1399677"
server_key_version: "encryptionkey@0"
I20260619 16:43:36.565917 4038 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.008s sys 0.000s
I20260619 16:43:36.568969 4053 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:43:36.570287 4038 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 16:43:36.570441 4038 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887385903558-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887385903558-388-0/minicluster-data/ts-0/wal
uuid: "3aae94a9a9da4e7abb6b2af5a691dfcf"
format_stamp: "Formatted at 2026-06-19 16:43:36 on dist-test-slave-wvz7"
server_key: "bd800c079077b20bdacc37e3fcf13957"
server_key_iv: "2614d5d88e25c500a6fc9872d1399677"
server_key_version: "encryptionkey@0"
I20260619 16:43:36.570632 4038 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887385903558-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887385903558-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887385903558-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:43:36.595506 4038 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:43:36.599933 4038 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:43:36.600221 4038 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:43:36.601184 4038 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:43:36.602523 4038 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:43:36.602600 4038 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:43:36.602699 4038 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:43:36.602756 4038 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:43:36.631893 4038 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:32941
I20260619 16:43:36.631938 4165 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:32941 every 8 connection(s)
I20260619 16:43:36.633039 4038 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887385903558-388-0/minicluster-data/ts-0/data/info.pb
I20260619 16:43:36.635603 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu as pid 4038
I20260619 16:43:36.635799 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887385903558-388-0/minicluster-data/ts-0/wal/instance
I20260619 16:43:36.636462 388 external_mini_cluster.cc:1428] Setting key 97aa262dba5d9821f0e61dc9d6db137d
Jun 19 16:43:36 dist-test-slave-wvz7 krb5kdc[3932](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887416, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260619 16:43:36.652667 4168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:36.635504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:45019 (local address 127.0.97.62:43381)
0619 16:43:36.635954 (+ 450us) server_negotiation.cc:206] Beginning negotiation
0619 16:43:36.635960 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:43:36.637082 (+ 1122us) server_negotiation.cc:407] Connection header received
0619 16:43:36.639227 (+ 2145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:43:36.639231 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:43:36.639293 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:43:36.639382 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:43:36.641235 (+ 1853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:36.642019 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:36.644012 (+ 1993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:36.644409 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:36.644516 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:43:36.647681 (+ 3165us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:43:36.647686 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:43:36.647689 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:43:36.647717 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:43:36.650005 (+ 2288us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:43:36.650631 (+ 626us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:43:36.650636 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:43:36.650638 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:43:36.650743 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:43:36.651111 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:43:36.651116 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:43:36.651119 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:43:36.651375 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:43:36.651498 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0619 16:43:36.652115 (+ 617us) server_negotiation.cc:299] Negotiation successful
0619 16:43:36.652326 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":330,"thread_start_us":209,"threads_started":1}
I20260619 16:43:36.653584 4169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:36.636087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43381 (local address 127.0.97.1:45019)
0619 16:43:36.636922 (+ 835us) negotiation.cc:107] Waiting for socket to connect
0619 16:43:36.636972 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0619 16:43:36.638936 (+ 1964us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:43:36.639640 (+ 704us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:43:36.639651 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:43:36.640343 (+ 692us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:43:36.641075 (+ 732us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:43:36.641097 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:43:36.642156 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:43:36.642160 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:43:36.643876 (+ 1716us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:43:36.643891 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:43:36.644608 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:43:36.644615 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:43:36.644844 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:43:36.645442 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:43:36.645463 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:43:36.647495 (+ 2032us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:43:36.650123 (+ 2628us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:43:36.650132 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:43:36.650148 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:43:36.650529 (+ 381us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:43:36.650843 (+ 314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:43:36.650848 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:43:36.650852 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:43:36.651016 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:43:36.651493 (+ 477us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:43:36.651500 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:43:36.651799 (+ 299us) client_negotiation.cc:769] Sending connection context
0619 16:43:36.652033 (+ 234us) client_negotiation.cc:240] Negotiation successful
0619 16:43:36.652522 (+ 489us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":555,"thread_start_us":319,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260619 16:43:36.655174 4166 heartbeater.cc:347] Connected to a master server at 127.0.97.62:43381
I20260619 16:43:36.655447 4166 heartbeater.cc:464] Registering TS with master...
I20260619 16:43:36.656098 4166 heartbeater.cc:511] Master 127.0.97.62:43381 requested a full tablet report, sending...
I20260619 16:43:36.658233 3977 ts_manager.cc:194] Registered new tserver with Master: 3aae94a9a9da4e7abb6b2af5a691dfcf (127.0.97.1:32941)
I20260619 16:43:36.660001 3977 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:45019
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887385903558-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887385903558-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260619 16:43:36.674777 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu
/tmp/dist-test-taskEs2EH8/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887385903558-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887385903558-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887385903558-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887385903558-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43381
--keytab_file=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887385903558-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40819
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887385903558-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887385903558-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887385903558-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:43:36.975755 4172 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:43:36.976063 4172 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:43:36.976202 4172 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:43:36.983402 4172 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:43:36.983526 4172 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:43:36.983675 4172 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260619 16:43:36.993309 4172 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40819
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887385903558-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887385903558-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887385903558-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887385903558-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43381
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887385903558-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:43:36.994647 4172 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:43:36.995946 4172 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:43:37.007956 4177 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:43:37.007917 4178 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:43:37.007987 4180 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:43:37.008942 4172 server_base.cc:1034] running on GCE node
I20260619 16:43:37.009532 4172 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:43:37.010440 4172 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:43:37.011674 4172 hybrid_clock.cc:648] HybridClock initialized: now 1781887417011616 us; error 43 us; skew 500 ppm
I20260619 16:43:37.012022 4172 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:43:37 dist-test-slave-wvz7 krb5kdc[3932](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887417, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:43:37.015139 4172 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260619 16:43:37.016578 4172 webserver.cc:460] Webserver started at http://127.0.97.2:37235/ using document root <none> and password file <none>
I20260619 16:43:37.017268 4172 fs_manager.cc:359] Metadata directory not provided
I20260619 16:43:37.017380 4172 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:43:37.017652 4172 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:43:37.020488 4172 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887385903558-388-0/minicluster-data/ts-1/data/instance:
uuid: "30dbd5f096ba47b1b23c129c150db9fe"
format_stamp: "Formatted at 2026-06-19 16:43:37 on dist-test-slave-wvz7"
server_key: "84e4bb03b8d3c908c39512c04208aefc"
server_key_iv: "7be7cd9a595dd3d6fbaa19f16c42315b"
server_key_version: "encryptionkey@0"
I20260619 16:43:37.021234 4172 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887385903558-388-0/minicluster-data/ts-1/wal/instance:
uuid: "30dbd5f096ba47b1b23c129c150db9fe"
format_stamp: "Formatted at 2026-06-19 16:43:37 on dist-test-slave-wvz7"
server_key: "84e4bb03b8d3c908c39512c04208aefc"
server_key_iv: "7be7cd9a595dd3d6fbaa19f16c42315b"
server_key_version: "encryptionkey@0"
I20260619 16:43:37.026265 4172 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260619 16:43:37.029837 4187 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:43:37.031283 4172 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 16:43:37.031445 4172 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887385903558-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887385903558-388-0/minicluster-data/ts-1/wal
uuid: "30dbd5f096ba47b1b23c129c150db9fe"
format_stamp: "Formatted at 2026-06-19 16:43:37 on dist-test-slave-wvz7"
server_key: "84e4bb03b8d3c908c39512c04208aefc"
server_key_iv: "7be7cd9a595dd3d6fbaa19f16c42315b"
server_key_version: "encryptionkey@0"
I20260619 16:43:37.031641 4172 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887385903558-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887385903558-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887385903558-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:43:37.043133 4172 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:43:37.046900 4172 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:43:37.047147 4172 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:43:37.048074 4172 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:43:37.049373 4172 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:43:37.049459 4172 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:43:37.049574 4172 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:43:37.049623 4172 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:43:37.078574 4172 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:41839
I20260619 16:43:37.078580 4299 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:41839 every 8 connection(s)
I20260619 16:43:37.080011 4172 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887385903558-388-0/minicluster-data/ts-1/data/info.pb
I20260619 16:43:37.090220 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu as pid 4172
I20260619 16:43:37.090405 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887385903558-388-0/minicluster-data/ts-1/wal/instance
I20260619 16:43:37.090785 388 external_mini_cluster.cc:1428] Setting key aece912992f9e322e9bf38ea682284d6
Jun 19 16:43:37 dist-test-slave-wvz7 krb5kdc[3932](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887417, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 16:43:37.100306 4168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:37.082430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:44625 (local address 127.0.97.62:43381)
0619 16:43:37.082628 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0619 16:43:37.082634 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:43:37.083855 (+ 1221us) server_negotiation.cc:407] Connection header received
0619 16:43:37.085572 (+ 1717us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:43:37.085578 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:43:37.085654 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:43:37.085784 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:43:37.087181 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:37.088267 (+ 1086us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:37.089773 (+ 1506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:37.090430 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:37.090555 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:43:37.094620 (+ 4065us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:43:37.094626 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:43:37.094631 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:43:37.094667 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:43:37.097530 (+ 2863us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:43:37.098290 (+ 760us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:43:37.098294 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:43:37.098296 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:43:37.098422 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:43:37.098845 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:43:37.098848 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:43:37.098850 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:43:37.099094 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:43:37.099202 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0619 16:43:37.099943 (+ 741us) server_negotiation.cc:299] Negotiation successful
0619 16:43:37.100096 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260619 16:43:37.101691 4302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:37.082939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43381 (local address 127.0.97.2:44625)
0619 16:43:37.083687 (+ 748us) negotiation.cc:107] Waiting for socket to connect
0619 16:43:37.083730 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0619 16:43:37.085322 (+ 1592us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:43:37.085967 (+ 645us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:43:37.085975 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:43:37.086485 (+ 510us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:43:37.087021 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:43:37.087038 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:43:37.088394 (+ 1356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:43:37.088397 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:43:37.089620 (+ 1223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:43:37.089632 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:43:37.090809 (+ 1177us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:43:37.090815 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:43:37.091089 (+ 274us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:43:37.092293 (+ 1204us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:43:37.092323 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:43:37.094397 (+ 2074us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:43:37.097660 (+ 3263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:43:37.097667 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:43:37.097680 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:43:37.098179 (+ 499us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:43:37.098512 (+ 333us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:43:37.098517 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:43:37.098520 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:43:37.098741 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:43:37.099205 (+ 464us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:43:37.099213 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:43:37.099596 (+ 383us) client_negotiation.cc:769] Sending connection context
0619 16:43:37.099878 (+ 282us) client_negotiation.cc:240] Negotiation successful
0619 16:43:37.100391 (+ 513us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":500,"thread_start_us":310,"threads_started":1}
I20260619 16:43:37.103071 4300 heartbeater.cc:347] Connected to a master server at 127.0.97.62:43381
I20260619 16:43:37.103379 4300 heartbeater.cc:464] Registering TS with master...
I20260619 16:43:37.104189 4300 heartbeater.cc:511] Master 127.0.97.62:43381 requested a full tablet report, sending...
I20260619 16:43:37.105769 3977 ts_manager.cc:194] Registered new tserver with Master: 30dbd5f096ba47b1b23c129c150db9fe (127.0.97.2:41839)
I20260619 16:43:37.106552 3977 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:44625
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887385903558-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887385903558-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260619 16:43:37.130584 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu
/tmp/dist-test-taskEs2EH8/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887385903558-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887385903558-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887385903558-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887385903558-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43381
--keytab_file=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887385903558-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40819
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887385903558-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887385903558-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887385903558-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:43:37.435786 4305 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:43:37.436075 4305 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:43:37.436211 4305 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:43:37.443482 4305 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:43:37.443609 4305 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:43:37.443760 4305 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260619 16:43:37.453351 4305 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40819
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887385903558-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887385903558-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887385903558-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887385903558-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43381
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887385903558-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:43:37.454636 4305 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:43:37.455960 4305 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:43:37.468309 4310 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:43:37.468353 4311 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:43:37.468437 4313 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:43:37.468938 4305 server_base.cc:1034] running on GCE node
I20260619 16:43:37.469506 4305 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:43:37.470418 4305 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:43:37.471624 4305 hybrid_clock.cc:648] HybridClock initialized: now 1781887417471571 us; error 47 us; skew 500 ppm
I20260619 16:43:37.471915 4305 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:43:37 dist-test-slave-wvz7 krb5kdc[3932](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887417, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:43:37.474922 4305 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260619 16:43:37.476279 4305 webserver.cc:460] Webserver started at http://127.0.97.3:46541/ using document root <none> and password file <none>
I20260619 16:43:37.476912 4305 fs_manager.cc:359] Metadata directory not provided
I20260619 16:43:37.477005 4305 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:43:37.477240 4305 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:43:37.479997 4305 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887385903558-388-0/minicluster-data/ts-2/data/instance:
uuid: "6fa862036b7e44529c6b01e3e926cc72"
format_stamp: "Formatted at 2026-06-19 16:43:37 on dist-test-slave-wvz7"
server_key: "e9ec5cce853540e7979f5895580c5aff"
server_key_iv: "6b7848ad8a7f73cefa432db6561ab3bd"
server_key_version: "encryptionkey@0"
I20260619 16:43:37.480686 4305 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887385903558-388-0/minicluster-data/ts-2/wal/instance:
uuid: "6fa862036b7e44529c6b01e3e926cc72"
format_stamp: "Formatted at 2026-06-19 16:43:37 on dist-test-slave-wvz7"
server_key: "e9ec5cce853540e7979f5895580c5aff"
server_key_iv: "6b7848ad8a7f73cefa432db6561ab3bd"
server_key_version: "encryptionkey@0"
I20260619 16:43:37.485625 4305 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.003s
I20260619 16:43:37.488874 4320 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:43:37.490236 4305 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 16:43:37.490372 4305 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887385903558-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887385903558-388-0/minicluster-data/ts-2/wal
uuid: "6fa862036b7e44529c6b01e3e926cc72"
format_stamp: "Formatted at 2026-06-19 16:43:37 on dist-test-slave-wvz7"
server_key: "e9ec5cce853540e7979f5895580c5aff"
server_key_iv: "6b7848ad8a7f73cefa432db6561ab3bd"
server_key_version: "encryptionkey@0"
I20260619 16:43:37.490545 4305 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887385903558-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887385903558-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887385903558-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:43:37.506475 4305 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:43:37.510710 4305 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:43:37.510954 4305 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:43:37.511850 4305 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:43:37.513131 4305 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:43:37.513206 4305 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:43:37.513294 4305 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:43:37.513335 4305 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:43:37.541639 4305 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:45713
I20260619 16:43:37.541669 4432 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:45713 every 8 connection(s)
I20260619 16:43:37.542829 4305 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887385903558-388-0/minicluster-data/ts-2/data/info.pb
I20260619 16:43:37.545898 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu as pid 4305
I20260619 16:43:37.546063 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887385903558-388-0/minicluster-data/ts-2/wal/instance
I20260619 16:43:37.546445 388 external_mini_cluster.cc:1428] Setting key c3c676e4af1f6acdbdb572bf722670d5
Jun 19 16:43:37 dist-test-slave-wvz7 krb5kdc[3932](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887417, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 16:43:37.561285 4168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:37.545085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:45975 (local address 127.0.97.62:43381)
0619 16:43:37.545274 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0619 16:43:37.545280 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:43:37.546491 (+ 1211us) server_negotiation.cc:407] Connection header received
0619 16:43:37.548540 (+ 2049us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:43:37.548545 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:43:37.548621 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:43:37.548721 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:43:37.550264 (+ 1543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:37.550982 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:37.552442 (+ 1460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:37.552852 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:37.552946 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:43:37.556001 (+ 3055us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:43:37.556005 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:43:37.556009 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:43:37.556039 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:43:37.558256 (+ 2217us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:43:37.558962 (+ 706us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:43:37.558966 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:43:37.558968 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:43:37.559087 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:43:37.559458 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:43:37.559462 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:43:37.559464 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:43:37.559728 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:43:37.559843 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0619 16:43:37.560746 (+ 903us) server_negotiation.cc:299] Negotiation successful
0619 16:43:37.560980 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260619 16:43:37.562222 4435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:37.545567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43381 (local address 127.0.97.3:45975)
0619 16:43:37.546330 (+ 763us) negotiation.cc:107] Waiting for socket to connect
0619 16:43:37.546380 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0619 16:43:37.548296 (+ 1916us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:43:37.548935 (+ 639us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:43:37.548945 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:43:37.549475 (+ 530us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:43:37.550104 (+ 629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:43:37.550122 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:43:37.551122 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:43:37.551126 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:43:37.552305 (+ 1179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:43:37.552316 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:43:37.553022 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:43:37.553026 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:43:37.553204 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:43:37.553813 (+ 609us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:43:37.553838 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:43:37.555822 (+ 1984us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:43:37.558393 (+ 2571us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:43:37.558399 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:43:37.558410 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:43:37.558836 (+ 426us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:43:37.559178 (+ 342us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:43:37.559181 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:43:37.559183 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:43:37.559351 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:43:37.560036 (+ 685us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:43:37.560042 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:43:37.560396 (+ 354us) client_negotiation.cc:769] Sending connection context
0619 16:43:37.560616 (+ 220us) client_negotiation.cc:240] Negotiation successful
0619 16:43:37.561053 (+ 437us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":507,"thread_start_us":326,"threads_started":1}
I20260619 16:43:37.563536 4433 heartbeater.cc:347] Connected to a master server at 127.0.97.62:43381
I20260619 16:43:37.563838 4433 heartbeater.cc:464] Registering TS with master...
I20260619 16:43:37.564500 4433 heartbeater.cc:511] Master 127.0.97.62:43381 requested a full tablet report, sending...
I20260619 16:43:37.565943 3977 ts_manager.cc:194] Registered new tserver with Master: 6fa862036b7e44529c6b01e3e926cc72 (127.0.97.3:45713)
I20260619 16:43:37.566701 3977 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:45975
I20260619 16:43:37.573446 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 19 16:43:37 dist-test-slave-wvz7 krb5kdc[3932](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887417, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 19 16:43:37 dist-test-slave-wvz7 krb5kdc[3932](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887417, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260619 16:43:37.617657 4440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:37.602802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56280 (local address 127.0.97.1:32941)
0619 16:43:37.603381 (+ 579us) server_negotiation.cc:206] Beginning negotiation
0619 16:43:37.603390 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 16:43:37.603421 (+ 31us) server_negotiation.cc:407] Connection header received
0619 16:43:37.603521 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:43:37.603529 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:43:37.603747 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:43:37.604000 (+ 253us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:43:37.605503 (+ 1503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:37.606769 (+ 1266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:37.607984 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:37.608540 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:37.608663 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:43:37.612277 (+ 3614us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:43:37.612287 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:43:37.612301 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:43:37.612341 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:43:37.615032 (+ 2691us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:43:37.615645 (+ 613us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:43:37.615652 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:43:37.615655 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:43:37.615771 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:43:37.616158 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:43:37.616163 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:43:37.616165 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:43:37.616503 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:43:37.616777 (+ 274us) server_negotiation.cc:1035] Waiting for connection context
0619 16:43:37.617073 (+ 296us) server_negotiation.cc:299] Negotiation successful
0619 16:43:37.617328 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":407,"thread_start_us":280,"threads_started":1}
Jun 19 16:43:37 dist-test-slave-wvz7 krb5kdc[3932](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887417, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 16:43:37.634485 4168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:37.621802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33248 (local address 127.0.97.62:43381)
0619 16:43:37.622015 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0619 16:43:37.622021 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:43:37.622148 (+ 127us) server_negotiation.cc:407] Connection header received
0619 16:43:37.622261 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:43:37.622266 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:43:37.622329 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:43:37.622400 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:43:37.623517 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:37.624246 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:37.625456 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:37.625938 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:37.626030 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:43:37.629463 (+ 3433us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:43:37.629468 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:43:37.629472 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:43:37.629503 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:43:37.631972 (+ 2469us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:43:37.632655 (+ 683us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:43:37.632659 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:43:37.632661 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:43:37.632785 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:43:37.633110 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:43:37.633113 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:43:37.633115 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:43:37.633454 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:43:37.633625 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0619 16:43:37.633967 (+ 342us) server_negotiation.cc:299] Negotiation successful
0619 16:43:37.634169 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
W20260619 16:43:37.635041 3977 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33248
I20260619 16:43:37.635978 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu pid 4038
I20260619 16:43:37.648710 4440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:37.637054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56296 (local address 127.0.97.1:32941)
0619 16:43:37.637241 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0619 16:43:37.637247 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:43:37.637376 (+ 129us) server_negotiation.cc:407] Connection header received
0619 16:43:37.637523 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:43:37.637527 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:43:37.637613 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:43:37.637686 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:43:37.639210 (+ 1524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:37.639899 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:37.641479 (+ 1580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:37.641958 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:37.642049 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:43:37.644195 (+ 2146us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:43:37.644200 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:43:37.644203 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:43:37.644231 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:43:37.646601 (+ 2370us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:43:37.647210 (+ 609us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:43:37.647214 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:43:37.647216 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:43:37.647358 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:43:37.647642 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:43:37.647645 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:43:37.647647 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:43:37.647887 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:43:37.648055 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0619 16:43:37.648254 (+ 199us) server_negotiation.cc:299] Negotiation successful
0619 16:43:37.648422 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260619 16:43:37.649258 4161 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:43:37.728070 4166 heartbeater.cc:503] Master 127.0.97.62:43381 was elected leader, sending a full tablet report...
I20260619 16:43:37.728546 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu with pid 4038
W20260619 16:43:37.741191 3964 connection.cc:535] server connection from 127.0.97.1:45019 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260619 16:43:37.741468 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu pid 4172
Jun 19 16:43:37 dist-test-slave-wvz7 krb5kdc[3932](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887417, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260619 16:43:37.756438 4445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:37.742299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37260 (local address 127.0.97.2:41839)
0619 16:43:37.742785 (+ 486us) server_negotiation.cc:206] Beginning negotiation
0619 16:43:37.742792 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:43:37.742817 (+ 25us) server_negotiation.cc:407] Connection header received
0619 16:43:37.742872 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:43:37.742878 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:43:37.743087 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:43:37.743219 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:43:37.744279 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:37.745252 (+ 973us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:37.746589 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:37.747175 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:37.747313 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:43:37.751267 (+ 3954us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:43:37.751279 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:43:37.751293 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:43:37.751332 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:43:37.753980 (+ 2648us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:43:37.754626 (+ 646us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:43:37.754632 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:43:37.754634 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:43:37.754748 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:43:37.755085 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:43:37.755088 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:43:37.755090 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:43:37.755424 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:43:37.755638 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0619 16:43:37.755896 (+ 258us) server_negotiation.cc:299] Negotiation successful
0619 16:43:37.756082 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":352,"thread_start_us":244,"threads_started":1}
I20260619 16:43:37.757642 4295 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:43:37.839526 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu with pid 4172
I20260619 16:43:37.851428 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu pid 4305
Jun 19 16:43:37 dist-test-slave-wvz7 krb5kdc[3932](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887417, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260619 16:43:37.866366 4447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:37.852390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37600 (local address 127.0.97.3:45713)
0619 16:43:37.852976 (+ 586us) server_negotiation.cc:206] Beginning negotiation
0619 16:43:37.852982 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:43:37.853008 (+ 26us) server_negotiation.cc:407] Connection header received
0619 16:43:37.853068 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:43:37.853074 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:43:37.853262 (+ 188us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:43:37.853425 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:43:37.854638 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:37.855612 (+ 974us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:37.856786 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:37.857314 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:37.857428 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:43:37.861385 (+ 3957us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:43:37.861394 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:43:37.861408 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:43:37.861438 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:43:37.863915 (+ 2477us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:43:37.864469 (+ 554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:43:37.864477 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:43:37.864479 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:43:37.864592 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:43:37.864916 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:43:37.864921 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:43:37.864925 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:43:37.865233 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:43:37.865521 (+ 288us) server_negotiation.cc:1035] Waiting for connection context
0619 16:43:37.865816 (+ 295us) server_negotiation.cc:299] Negotiation successful
0619 16:43:37.866055 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":451,"thread_start_us":292,"threads_started":1}
I20260619 16:43:37.867570 4428 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:43:37.950855 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu with pid 4305
I20260619 16:43:37.962416 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu pid 3945
I20260619 16:43:37.963536 4007 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:43:38.063771 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu with pid 3945
2026-06-19T16:43:38Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (3550 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887385903558-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 16:43:38 dist-test-slave-wvz7 krb5kdc[4452](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 16:43:38 dist-test-slave-wvz7 krb5kdc[4452](info): set up 2 sockets
Jun 19 16:43:38 dist-test-slave-wvz7 krb5kdc[4452](info): commencing operation
krb5kdc: starting...
W20260619 16:43:39.061954 388 mini_kdc.cc:121] Time spent starting KDC: real 0.963s user 0.007s sys 0.010s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887385903558-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887385903558-388-0/krb5kdc/test-user.keytab.
Jun 19 16:43:39 dist-test-slave-wvz7 krb5kdc[4452](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887419, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T16:43:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T16:43:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887385903558-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887385903558-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260619 16:43:39.239323 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu
/tmp/dist-test-taskEs2EH8/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887385903558-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887385903558-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887385903558-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887385903558-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:32779
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887385903558-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:46015
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:32779
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887385903558-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887385903558-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887385903558-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:43:39.542418 4465 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:43:39.542727 4465 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:43:39.542847 4465 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:43:39.549805 4465 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:43:39.549903 4465 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:43:39.549974 4465 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:43:39.550047 4465 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:43:39.550112 4465 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:43:39.559561 4465 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46015
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887385903558-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887385903558-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:32779
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887385903558-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:32779
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887385903558-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887385903558-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:43:39.560865 4465 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:43:39.562171 4465 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:43:39.572903 4473 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:43:39.572968 4471 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:43:39.573007 4470 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:43:39.574041 4465 server_base.cc:1034] running on GCE node
I20260619 16:43:39.574774 4465 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:43:39.576006 4465 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:43:39.577219 4465 hybrid_clock.cc:648] HybridClock initialized: now 1781887419577174 us; error 41 us; skew 500 ppm
I20260619 16:43:39.577534 4465 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:43:39 dist-test-slave-wvz7 krb5kdc[4452](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887419, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:43:39.580396 4465 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260619 16:43:39.581413 4465 fs_manager.cc:359] Metadata directory not provided
I20260619 16:43:39.581497 4465 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:43:39.581768 4465 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:43:39.584643 4465 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887385903558-388-0/minicluster-data/master-0/data/instance:
uuid: "f993ca2899fd4e779ab2d78a4697b890"
format_stamp: "Formatted at 2026-06-19 16:43:39 on dist-test-slave-wvz7"
server_key: "f9a7d35ef568327f6551fa707441e9f9"
server_key_iv: "cf5cd78908f41d7f63022189533bb29e"
server_key_version: "encryptionkey@0"
I20260619 16:43:39.585351 4465 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887385903558-388-0/minicluster-data/master-0/wal/instance:
uuid: "f993ca2899fd4e779ab2d78a4697b890"
format_stamp: "Formatted at 2026-06-19 16:43:39 on dist-test-slave-wvz7"
server_key: "f9a7d35ef568327f6551fa707441e9f9"
server_key_iv: "cf5cd78908f41d7f63022189533bb29e"
server_key_version: "encryptionkey@0"
I20260619 16:43:39.590122 4465 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.004s
I20260619 16:43:39.593688 4479 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:43:39.595091 4465 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 16:43:39.595258 4465 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887385903558-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887385903558-388-0/minicluster-data/master-0/wal
uuid: "f993ca2899fd4e779ab2d78a4697b890"
format_stamp: "Formatted at 2026-06-19 16:43:39 on dist-test-slave-wvz7"
server_key: "f9a7d35ef568327f6551fa707441e9f9"
server_key_iv: "cf5cd78908f41d7f63022189533bb29e"
server_key_version: "encryptionkey@0"
I20260619 16:43:39.595453 4465 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887385903558-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887385903558-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887385903558-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:43:39.620651 4465 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:43:39.624615 4465 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:43:39.624857 4465 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:43:39.640836 4465 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:32779
I20260619 16:43:39.640812 4530 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:32779 every 8 connection(s)
I20260619 16:43:39.641716 4465 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887385903558-388-0/minicluster-data/master-0/data/info.pb
I20260619 16:43:39.645076 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu as pid 4465
I20260619 16:43:39.645179 4531 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:43:39.645267 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887385903558-388-0/minicluster-data/master-0/wal/instance
I20260619 16:43:39.645900 388 external_mini_cluster.cc:1428] Setting key d38df974df4218554f7bd05a5e6bc3d3
I20260619 16:43:39.658216 4531 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f993ca2899fd4e779ab2d78a4697b890: Bootstrap starting.
Jun 19 16:43:39 dist-test-slave-wvz7 krb5kdc[4452](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887419, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 16:43:39.661149 4531 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f993ca2899fd4e779ab2d78a4697b890: Neither blocks nor log segments found. Creating new log.
I20260619 16:43:39.662418 4531 log.cc:824] T 00000000000000000000000000000000 P f993ca2899fd4e779ab2d78a4697b890: Log is configured to *not* fsync() on all Append() calls
I20260619 16:43:39.665594 4531 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f993ca2899fd4e779ab2d78a4697b890: No bootstrap required, opened a new log
I20260619 16:43:39.668442 4534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:39.647650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56104 (local address 127.0.97.62:32779)
0619 16:43:39.648430 (+ 780us) server_negotiation.cc:206] Beginning negotiation
0619 16:43:39.648444 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 16:43:39.648494 (+ 50us) server_negotiation.cc:407] Connection header received
0619 16:43:39.651040 (+ 2546us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:43:39.651068 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:43:39.651578 (+ 510us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:43:39.651990 (+ 412us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:43:39.653188 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:39.655119 (+ 1931us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:39.656258 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:39.657235 (+ 977us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:39.657420 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:43:39.660846 (+ 3426us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:43:39.660856 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:43:39.660875 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:43:39.660915 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:43:39.664521 (+ 3606us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:43:39.665339 (+ 818us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:43:39.665345 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:43:39.665351 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:43:39.665509 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:43:39.665941 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:43:39.665945 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:43:39.665947 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:43:39.666465 (+ 518us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:43:39.666662 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0619 16:43:39.667079 (+ 417us) server_negotiation.cc:299] Negotiation successful
0619 16:43:39.667424 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":541,"thread_start_us":296,"threads_started":1}
I20260619 16:43:39.673125 4531 raft_consensus.cc:348] T 00000000000000000000000000000000 P f993ca2899fd4e779ab2d78a4697b890 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f993ca2899fd4e779ab2d78a4697b890" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 32779 } }
I20260619 16:43:39.673425 4531 raft_consensus.cc:374] T 00000000000000000000000000000000 P f993ca2899fd4e779ab2d78a4697b890 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:43:39.673514 4531 raft_consensus.cc:729] T 00000000000000000000000000000000 P f993ca2899fd4e779ab2d78a4697b890 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f993ca2899fd4e779ab2d78a4697b890, State: Initialized, Role: FOLLOWER
I20260619 16:43:39.674032 4531 consensus_queue.cc:260] T 00000000000000000000000000000000 P f993ca2899fd4e779ab2d78a4697b890 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f993ca2899fd4e779ab2d78a4697b890" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 32779 } }
I20260619 16:43:39.674230 4531 raft_consensus.cc:388] T 00000000000000000000000000000000 P f993ca2899fd4e779ab2d78a4697b890 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:43:39.674345 4531 raft_consensus.cc:482] T 00000000000000000000000000000000 P f993ca2899fd4e779ab2d78a4697b890 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:43:39.674497 4531 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f993ca2899fd4e779ab2d78a4697b890 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:43:39.676476 4531 raft_consensus.cc:504] T 00000000000000000000000000000000 P f993ca2899fd4e779ab2d78a4697b890 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f993ca2899fd4e779ab2d78a4697b890" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 32779 } }
I20260619 16:43:39.676911 4531 leader_election.cc:302] T 00000000000000000000000000000000 P f993ca2899fd4e779ab2d78a4697b890 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f993ca2899fd4e779ab2d78a4697b890; no voters:
I20260619 16:43:39.677400 4531 leader_election.cc:288] T 00000000000000000000000000000000 P f993ca2899fd4e779ab2d78a4697b890 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 16:43:39.677481 4536 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f993ca2899fd4e779ab2d78a4697b890 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:43:39.678102 4536 raft_consensus.cc:686] T 00000000000000000000000000000000 P f993ca2899fd4e779ab2d78a4697b890 [term 1 LEADER]: Becoming Leader. State: Replica: f993ca2899fd4e779ab2d78a4697b890, State: Running, Role: LEADER
I20260619 16:43:39.678544 4536 consensus_queue.cc:237] T 00000000000000000000000000000000 P f993ca2899fd4e779ab2d78a4697b890 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f993ca2899fd4e779ab2d78a4697b890" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 32779 } }
I20260619 16:43:39.679158 4531 sys_catalog.cc:564] T 00000000000000000000000000000000 P f993ca2899fd4e779ab2d78a4697b890 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:43:39.681044 4537 sys_catalog.cc:455] T 00000000000000000000000000000000 P f993ca2899fd4e779ab2d78a4697b890 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f993ca2899fd4e779ab2d78a4697b890" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f993ca2899fd4e779ab2d78a4697b890" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 32779 } } }
I20260619 16:43:39.681412 4537 sys_catalog.cc:458] T 00000000000000000000000000000000 P f993ca2899fd4e779ab2d78a4697b890 [sys.catalog]: This master's current role is: LEADER
I20260619 16:43:39.682657 4538 sys_catalog.cc:455] T 00000000000000000000000000000000 P f993ca2899fd4e779ab2d78a4697b890 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f993ca2899fd4e779ab2d78a4697b890. Latest consensus state: current_term: 1 leader_uuid: "f993ca2899fd4e779ab2d78a4697b890" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f993ca2899fd4e779ab2d78a4697b890" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 32779 } } }
I20260619 16:43:39.682956 4538 sys_catalog.cc:458] T 00000000000000000000000000000000 P f993ca2899fd4e779ab2d78a4697b890 [sys.catalog]: This master's current role is: LEADER
I20260619 16:43:39.687156 4545 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:43:39.691176 4545 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:43:39.692095 4553 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260619 16:43:39.699250 4545 catalog_manager.cc:1347] Generated new cluster ID: dce44023319a4a71905df165c3fe3551
I20260619 16:43:39.699354 4545 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:43:39.714475 4545 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 16:43:39.715737 4545 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:43:39.732432 4545 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f993ca2899fd4e779ab2d78a4697b890: Generated new TSK 0
I20260619 16:43:39.733317 4545 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887385903558-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887385903558-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260619 16:43:39.786813 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu
/tmp/dist-test-taskEs2EH8/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887385903558-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887385903558-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887385903558-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887385903558-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:32779
--keytab_file=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887385903558-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:46015
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887385903558-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887385903558-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887385903558-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:43:40.096063 4557 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:43:40.096341 4557 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:43:40.096446 4557 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:43:40.103672 4557 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:43:40.103777 4557 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:43:40.103892 4557 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260619 16:43:40.113437 4557 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46015
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887385903558-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887385903558-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887385903558-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887385903558-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:32779
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887385903558-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:43:40.114660 4557 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:43:40.115847 4557 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:43:40.127790 4563 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:43:40.127774 4562 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:43:40.128316 4565 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:43:40.128747 4557 server_base.cc:1034] running on GCE node
I20260619 16:43:40.129315 4557 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:43:40.130252 4557 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:43:40.131448 4557 hybrid_clock.cc:648] HybridClock initialized: now 1781887420131400 us; error 37 us; skew 500 ppm
I20260619 16:43:40.131778 4557 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:43:40 dist-test-slave-wvz7 krb5kdc[4452](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887420, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:43:40.134835 4557 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260619 16:43:40.135895 4557 fs_manager.cc:359] Metadata directory not provided
I20260619 16:43:40.135987 4557 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:43:40.136270 4557 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:43:40.138990 4557 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887385903558-388-0/minicluster-data/ts-0/data/instance:
uuid: "57251729b4cb4277ac2cbe7a678f72df"
format_stamp: "Formatted at 2026-06-19 16:43:40 on dist-test-slave-wvz7"
server_key: "319baaccc7f92e2c9a7a78294802e4ce"
server_key_iv: "9c51149d66b95b310e68e0f433205f72"
server_key_version: "encryptionkey@0"
I20260619 16:43:40.139755 4557 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887385903558-388-0/minicluster-data/ts-0/wal/instance:
uuid: "57251729b4cb4277ac2cbe7a678f72df"
format_stamp: "Formatted at 2026-06-19 16:43:40 on dist-test-slave-wvz7"
server_key: "319baaccc7f92e2c9a7a78294802e4ce"
server_key_iv: "9c51149d66b95b310e68e0f433205f72"
server_key_version: "encryptionkey@0"
I20260619 16:43:40.144804 4557 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260619 16:43:40.148221 4571 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:43:40.149657 4557 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 16:43:40.149881 4557 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887385903558-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887385903558-388-0/minicluster-data/ts-0/wal
uuid: "57251729b4cb4277ac2cbe7a678f72df"
format_stamp: "Formatted at 2026-06-19 16:43:40 on dist-test-slave-wvz7"
server_key: "319baaccc7f92e2c9a7a78294802e4ce"
server_key_iv: "9c51149d66b95b310e68e0f433205f72"
server_key_version: "encryptionkey@0"
I20260619 16:43:40.150077 4557 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887385903558-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887385903558-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887385903558-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:43:40.166839 4557 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:43:40.171072 4557 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:43:40.171305 4557 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:43:40.172192 4557 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:43:40.173478 4557 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:43:40.173569 4557 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:43:40.173666 4557 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:43:40.173765 4557 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:43:40.202980 4557 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:45727
I20260619 16:43:40.203012 4683 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:45727 every 8 connection(s)
I20260619 16:43:40.203778 4557 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887385903558-388-0/minicluster-data/ts-0/data/info.pb
I20260619 16:43:40.213794 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu as pid 4557
I20260619 16:43:40.213984 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887385903558-388-0/minicluster-data/ts-0/wal/instance
I20260619 16:43:40.214604 388 external_mini_cluster.cc:1428] Setting key 1bb180e6edd30406b05052036228cee4
Jun 19 16:43:40 dist-test-slave-wvz7 krb5kdc[4452](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887420, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 16:43:40.221558 4686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:40.205915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:58437 (local address 127.0.97.62:32779)
0619 16:43:40.206325 (+ 410us) server_negotiation.cc:206] Beginning negotiation
0619 16:43:40.206329 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:43:40.207208 (+ 879us) server_negotiation.cc:407] Connection header received
0619 16:43:40.208891 (+ 1683us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:43:40.208897 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:43:40.208972 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:43:40.209065 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:43:40.210525 (+ 1460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:40.211237 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:40.212676 (+ 1439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:40.213057 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:40.213171 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:43:40.216553 (+ 3382us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:43:40.216557 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:43:40.216561 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:43:40.216591 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:43:40.218874 (+ 2283us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:43:40.219502 (+ 628us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:43:40.219506 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:43:40.219509 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:43:40.219616 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:43:40.219982 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:43:40.219987 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:43:40.219989 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:43:40.220246 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:43:40.220371 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0619 16:43:40.220996 (+ 625us) server_negotiation.cc:299] Negotiation successful
0619 16:43:40.221220 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":317,"thread_start_us":222,"threads_started":1}
I20260619 16:43:40.222431 4687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:40.206400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:32779 (local address 127.0.97.1:58437)
0619 16:43:40.207047 (+ 647us) negotiation.cc:107] Waiting for socket to connect
0619 16:43:40.207089 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0619 16:43:40.208646 (+ 1557us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:43:40.209284 (+ 638us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:43:40.209292 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:43:40.209840 (+ 548us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:43:40.210367 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:43:40.210385 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:43:40.211365 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:43:40.211368 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:43:40.212545 (+ 1177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:43:40.212557 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:43:40.213253 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:43:40.213260 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:43:40.213491 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:43:40.214110 (+ 619us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:43:40.214139 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:43:40.216343 (+ 2204us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:43:40.218991 (+ 2648us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:43:40.218999 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:43:40.219014 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:43:40.219389 (+ 375us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:43:40.219710 (+ 321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:43:40.219715 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:43:40.219719 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:43:40.219877 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:43:40.220357 (+ 480us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:43:40.220364 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:43:40.220667 (+ 303us) client_negotiation.cc:769] Sending connection context
0619 16:43:40.220881 (+ 214us) client_negotiation.cc:240] Negotiation successful
0619 16:43:40.221334 (+ 453us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":434,"thread_start_us":268,"threads_started":1}
I20260619 16:43:40.223810 4684 heartbeater.cc:347] Connected to a master server at 127.0.97.62:32779
I20260619 16:43:40.224061 4684 heartbeater.cc:464] Registering TS with master...
I20260619 16:43:40.224663 4684 heartbeater.cc:511] Master 127.0.97.62:32779 requested a full tablet report, sending...
I20260619 16:43:40.226560 4496 ts_manager.cc:194] Registered new tserver with Master: 57251729b4cb4277ac2cbe7a678f72df (127.0.97.1:45727)
I20260619 16:43:40.227931 4496 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:58437
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887385903558-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887385903558-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260619 16:43:40.250852 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu
/tmp/dist-test-taskEs2EH8/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887385903558-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887385903558-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887385903558-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887385903558-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:32779
--keytab_file=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887385903558-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:46015
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887385903558-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887385903558-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887385903558-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:43:40.550650 4690 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:43:40.550915 4690 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:43:40.551007 4690 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:43:40.557950 4690 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:43:40.558050 4690 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:43:40.558161 4690 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260619 16:43:40.567498 4690 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46015
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887385903558-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887385903558-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887385903558-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887385903558-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:32779
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887385903558-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:43:40.568691 4690 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:43:40.569942 4690 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:43:40.581475 4695 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:43:40.581509 4696 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:43:40.582011 4698 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:43:40.582387 4690 server_base.cc:1034] running on GCE node
I20260619 16:43:40.582960 4690 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:43:40.583854 4690 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:43:40.585053 4690 hybrid_clock.cc:648] HybridClock initialized: now 1781887420584999 us; error 53 us; skew 500 ppm
I20260619 16:43:40.585359 4690 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:43:40 dist-test-slave-wvz7 krb5kdc[4452](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887420, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:43:40.588279 4690 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260619 16:43:40.589299 4690 fs_manager.cc:359] Metadata directory not provided
I20260619 16:43:40.589392 4690 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:43:40.589671 4690 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:43:40.592377 4690 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887385903558-388-0/minicluster-data/ts-1/data/instance:
uuid: "9fe332844aef4d7badc14dea7b7266b9"
format_stamp: "Formatted at 2026-06-19 16:43:40 on dist-test-slave-wvz7"
server_key: "12bbcd55ffab69315150cf3118600bca"
server_key_iv: "3fc5029d2d3de95b97138c54694c60a9"
server_key_version: "encryptionkey@0"
I20260619 16:43:40.593123 4690 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887385903558-388-0/minicluster-data/ts-1/wal/instance:
uuid: "9fe332844aef4d7badc14dea7b7266b9"
format_stamp: "Formatted at 2026-06-19 16:43:40 on dist-test-slave-wvz7"
server_key: "12bbcd55ffab69315150cf3118600bca"
server_key_iv: "3fc5029d2d3de95b97138c54694c60a9"
server_key_version: "encryptionkey@0"
I20260619 16:43:40.598016 4690 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260619 16:43:40.601227 4704 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:43:40.602522 4690 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 16:43:40.602671 4690 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887385903558-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887385903558-388-0/minicluster-data/ts-1/wal
uuid: "9fe332844aef4d7badc14dea7b7266b9"
format_stamp: "Formatted at 2026-06-19 16:43:40 on dist-test-slave-wvz7"
server_key: "12bbcd55ffab69315150cf3118600bca"
server_key_iv: "3fc5029d2d3de95b97138c54694c60a9"
server_key_version: "encryptionkey@0"
I20260619 16:43:40.602860 4690 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887385903558-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887385903558-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887385903558-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:43:40.618407 4690 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:43:40.622206 4690 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:43:40.622444 4690 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:43:40.623322 4690 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:43:40.624553 4690 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:43:40.624640 4690 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:43:40.624740 4690 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:43:40.624809 4690 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:43:40.653628 4690 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:39019
I20260619 16:43:40.653707 4816 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:39019 every 8 connection(s)
I20260619 16:43:40.654474 4690 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887385903558-388-0/minicluster-data/ts-1/data/info.pb
I20260619 16:43:40.656188 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu as pid 4690
I20260619 16:43:40.656381 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887385903558-388-0/minicluster-data/ts-1/wal/instance
I20260619 16:43:40.656888 388 external_mini_cluster.cc:1428] Setting key 3891e77fd581431b7b7ae51b324a21e0
Jun 19 16:43:40 dist-test-slave-wvz7 krb5kdc[4452](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887420, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260619 16:43:40.674021 4686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:40.656928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:60591 (local address 127.0.97.62:32779)
0619 16:43:40.657098 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0619 16:43:40.657104 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:43:40.658405 (+ 1301us) server_negotiation.cc:407] Connection header received
0619 16:43:40.660174 (+ 1769us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:43:40.660180 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:43:40.660266 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:43:40.660349 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:43:40.661798 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:40.662493 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:40.664550 (+ 2057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:40.664933 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:40.665023 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:43:40.668657 (+ 3634us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:43:40.668661 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:43:40.668664 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:43:40.668694 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:43:40.671121 (+ 2427us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:43:40.671934 (+ 813us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:43:40.671937 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:43:40.671940 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:43:40.672050 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:43:40.672454 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:43:40.672457 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:43:40.672459 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:43:40.672678 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:43:40.672776 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0619 16:43:40.673558 (+ 782us) server_negotiation.cc:299] Negotiation successful
0619 16:43:40.673711 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260619 16:43:40.675105 4819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:40.657499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:32779 (local address 127.0.97.2:60591)
0619 16:43:40.658233 (+ 734us) negotiation.cc:107] Waiting for socket to connect
0619 16:43:40.658286 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0619 16:43:40.659931 (+ 1645us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:43:40.660577 (+ 646us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:43:40.660585 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:43:40.661084 (+ 499us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:43:40.661611 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:43:40.661627 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:43:40.662631 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:43:40.662635 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:43:40.664397 (+ 1762us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:43:40.664411 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:43:40.665126 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:43:40.665132 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:43:40.665377 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:43:40.666035 (+ 658us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:43:40.666065 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:43:40.668466 (+ 2401us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:43:40.671251 (+ 2785us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:43:40.671259 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:43:40.671277 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:43:40.671840 (+ 563us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:43:40.672139 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:43:40.672144 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:43:40.672147 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:43:40.672364 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:43:40.672795 (+ 431us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:43:40.672803 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:43:40.673204 (+ 401us) client_negotiation.cc:769] Sending connection context
0619 16:43:40.673518 (+ 314us) client_negotiation.cc:240] Negotiation successful
0619 16:43:40.674058 (+ 540us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":484,"thread_start_us":299,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
I20260619 16:43:40.676442 4817 heartbeater.cc:347] Connected to a master server at 127.0.97.62:32779
I20260619 16:43:40.676746 4817 heartbeater.cc:464] Registering TS with master...
I20260619 16:43:40.677495 4817 heartbeater.cc:511] Master 127.0.97.62:32779 requested a full tablet report, sending...
I20260619 16:43:40.678994 4496 ts_manager.cc:194] Registered new tserver with Master: 9fe332844aef4d7badc14dea7b7266b9 (127.0.97.2:39019)
I20260619 16:43:40.679965 4496 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:60591
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887385903558-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887385903558-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260619 16:43:40.696651 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu
/tmp/dist-test-taskEs2EH8/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887385903558-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887385903558-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887385903558-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887385903558-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:32779
--keytab_file=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887385903558-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:46015
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887385903558-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887385903558-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887385903558-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:43:41.000038 4822 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:43:41.000306 4822 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:43:41.000432 4822 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:43:41.007627 4822 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:43:41.007761 4822 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:43:41.007902 4822 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260619 16:43:41.017639 4822 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46015
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887385903558-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887385903558-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887385903558-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887385903558-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:32779
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887385903558-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:43:41.018996 4822 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:43:41.020292 4822 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:43:41.032791 4827 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:43:41.032826 4828 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:43:41.032851 4830 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:43:41.033357 4822 server_base.cc:1034] running on GCE node
I20260619 16:43:41.033995 4822 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:43:41.034953 4822 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:43:41.036157 4822 hybrid_clock.cc:648] HybridClock initialized: now 1781887421036106 us; error 41 us; skew 500 ppm
I20260619 16:43:41.036464 4822 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:43:41 dist-test-slave-wvz7 krb5kdc[4452](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887421, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:43:41.043519 4822 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260619 16:43:41.044698 4822 fs_manager.cc:359] Metadata directory not provided
I20260619 16:43:41.044804 4822 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:43:41.045081 4822 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:43:41.047870 4822 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887385903558-388-0/minicluster-data/ts-2/data/instance:
uuid: "42601d81443840cb88ea5a2c2ba6a1d5"
format_stamp: "Formatted at 2026-06-19 16:43:41 on dist-test-slave-wvz7"
server_key: "39d32f07a70ef27f4082ca1d02b2d2cf"
server_key_iv: "eab53a225412c8dd1af048a29ff15432"
server_key_version: "encryptionkey@0"
I20260619 16:43:41.048621 4822 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887385903558-388-0/minicluster-data/ts-2/wal/instance:
uuid: "42601d81443840cb88ea5a2c2ba6a1d5"
format_stamp: "Formatted at 2026-06-19 16:43:41 on dist-test-slave-wvz7"
server_key: "39d32f07a70ef27f4082ca1d02b2d2cf"
server_key_iv: "eab53a225412c8dd1af048a29ff15432"
server_key_version: "encryptionkey@0"
I20260619 16:43:41.053511 4822 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260619 16:43:41.056857 4836 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:43:41.058171 4822 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 16:43:41.058312 4822 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887385903558-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887385903558-388-0/minicluster-data/ts-2/wal
uuid: "42601d81443840cb88ea5a2c2ba6a1d5"
format_stamp: "Formatted at 2026-06-19 16:43:41 on dist-test-slave-wvz7"
server_key: "39d32f07a70ef27f4082ca1d02b2d2cf"
server_key_iv: "eab53a225412c8dd1af048a29ff15432"
server_key_version: "encryptionkey@0"
I20260619 16:43:41.058496 4822 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887385903558-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887385903558-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887385903558-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:43:41.083313 4822 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:43:41.087507 4822 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:43:41.087774 4822 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:43:41.088693 4822 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:43:41.090006 4822 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:43:41.090096 4822 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:43:41.090196 4822 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:43:41.090265 4822 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:43:41.119136 4822 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:34555
I20260619 16:43:41.119211 4948 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:34555 every 8 connection(s)
I20260619 16:43:41.119930 4822 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887385903558-388-0/minicluster-data/ts-2/data/info.pb
I20260619 16:43:41.123250 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu as pid 4822
I20260619 16:43:41.123423 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887385903558-388-0/minicluster-data/ts-2/wal/instance
I20260619 16:43:41.123875 388 external_mini_cluster.cc:1428] Setting key 13f9052d8d24d8556aa8e0372898f8e5
Jun 19 16:43:41 dist-test-slave-wvz7 krb5kdc[4452](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887421, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 16:43:41.139689 4686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:41.122199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:47351 (local address 127.0.97.62:32779)
0619 16:43:41.122402 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0619 16:43:41.122408 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:43:41.123700 (+ 1292us) server_negotiation.cc:407] Connection header received
0619 16:43:41.126018 (+ 2318us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:43:41.126024 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:43:41.126100 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:43:41.126211 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:43:41.127865 (+ 1654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:41.128583 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:41.130339 (+ 1756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:41.130733 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:41.130822 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:43:41.134258 (+ 3436us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:43:41.134262 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:43:41.134266 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:43:41.134296 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:43:41.136587 (+ 2291us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:43:41.137249 (+ 662us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:43:41.137253 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:43:41.137255 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:43:41.137364 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:43:41.138049 (+ 685us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:43:41.138054 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:43:41.138057 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:43:41.138397 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:43:41.138538 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0619 16:43:41.139276 (+ 738us) server_negotiation.cc:299] Negotiation successful
0619 16:43:41.139445 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260619 16:43:41.140533 4951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:41.122722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:32779 (local address 127.0.97.3:47351)
0619 16:43:41.123495 (+ 773us) negotiation.cc:107] Waiting for socket to connect
0619 16:43:41.123544 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0619 16:43:41.125727 (+ 2183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:43:41.126441 (+ 714us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:43:41.126454 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:43:41.127121 (+ 667us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:43:41.127714 (+ 593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:43:41.127732 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:43:41.128807 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:43:41.128811 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:43:41.130219 (+ 1408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:43:41.130232 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:43:41.130965 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:43:41.130972 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:43:41.131180 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:43:41.131829 (+ 649us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:43:41.131854 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:43:41.134071 (+ 2217us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:43:41.136726 (+ 2655us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:43:41.136733 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:43:41.136746 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:43:41.137133 (+ 387us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:43:41.137457 (+ 324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:43:41.137461 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:43:41.137463 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:43:41.137687 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:43:41.138519 (+ 832us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:43:41.138524 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:43:41.138907 (+ 383us) client_negotiation.cc:769] Sending connection context
0619 16:43:41.139128 (+ 221us) client_negotiation.cc:240] Negotiation successful
0619 16:43:41.139505 (+ 377us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":484,"thread_start_us":324,"threads_started":1}
I20260619 16:43:41.141753 4949 heartbeater.cc:347] Connected to a master server at 127.0.97.62:32779
I20260619 16:43:41.142007 4949 heartbeater.cc:464] Registering TS with master...
I20260619 16:43:41.142566 4949 heartbeater.cc:511] Master 127.0.97.62:32779 requested a full tablet report, sending...
I20260619 16:43:41.143814 4496 ts_manager.cc:194] Registered new tserver with Master: 42601d81443840cb88ea5a2c2ba6a1d5 (127.0.97.3:34555)
I20260619 16:43:41.144582 4496 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:47351
I20260619 16:43:41.150717 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 16:43:41.166579 4686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:41.154930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56114 (local address 127.0.97.62:32779)
0619 16:43:41.155091 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0619 16:43:41.155095 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:43:41.155531 (+ 436us) server_negotiation.cc:407] Connection header received
0619 16:43:41.155687 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:43:41.155691 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:43:41.155767 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:43:41.155846 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:43:41.157171 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:41.157911 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:41.159135 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:41.159548 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:41.159654 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:43:41.161816 (+ 2162us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:43:41.161821 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:43:41.161824 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:43:41.161859 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:43:41.164179 (+ 2320us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:43:41.164763 (+ 584us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:43:41.164769 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:43:41.164772 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:43:41.164891 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:43:41.165214 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:43:41.165219 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:43:41.165223 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:43:41.165546 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:43:41.165791 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0619 16:43:41.166081 (+ 290us) server_negotiation.cc:299] Negotiation successful
0619 16:43:41.166298 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260619 16:43:41.170419 4496 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56114:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 16:43:41.177088 4496 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 16:43:41.207401 4961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:41.189087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:39019 (local address 127.0.0.1:53280)
0619 16:43:41.189862 (+ 775us) negotiation.cc:107] Waiting for socket to connect
0619 16:43:41.189897 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0619 16:43:41.190056 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:43:41.197483 (+ 7427us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:43:41.197492 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:43:41.197540 (+ 48us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:43:41.198020 (+ 480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:43:41.198033 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:43:41.201000 (+ 2967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:43:41.201005 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:43:41.204285 (+ 3280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:43:41.204300 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:43:41.206344 (+ 2044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:43:41.206351 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:43:41.206539 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:43:41.206592 (+ 53us) client_negotiation.cc:769] Sending connection context
0619 16:43:41.206715 (+ 123us) client_negotiation.cc:240] Negotiation successful
0619 16:43:41.206992 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":502,"thread_start_us":656,"threads_started":1}
I20260619 16:43:41.208824 4962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:41.189177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53280 (local address 127.0.97.2:39019)
0619 16:43:41.190594 (+ 1417us) server_negotiation.cc:206] Beginning negotiation
0619 16:43:41.190604 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 16:43:41.190633 (+ 29us) server_negotiation.cc:407] Connection header received
0619 16:43:41.190732 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:43:41.190739 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:43:41.191007 (+ 268us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:43:41.191202 (+ 195us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:43:41.198606 (+ 7404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:41.200269 (+ 1663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:41.204432 (+ 4163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:41.206095 (+ 1663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:41.208088 (+ 1993us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:43:41.208281 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0619 16:43:41.208353 (+ 72us) server_negotiation.cc:299] Negotiation successful
0619 16:43:41.208507 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1252,"thread_start_us":2171,"threads_started":1}
I20260619 16:43:41.209976 4963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:41.189363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45727 (local address 127.0.0.1:47318)
0619 16:43:41.201503 (+ 12140us) negotiation.cc:107] Waiting for socket to connect
0619 16:43:41.201521 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 16:43:41.201652 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:43:41.202654 (+ 1002us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:43:41.202660 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:43:41.202686 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:43:41.203129 (+ 443us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:43:41.203140 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:43:41.205139 (+ 1999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:43:41.205154 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:43:41.207102 (+ 1948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:43:41.207123 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:43:41.209107 (+ 1984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:43:41.209114 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:43:41.209288 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:43:41.209381 (+ 93us) client_negotiation.cc:769] Sending connection context
0619 16:43:41.209457 (+ 76us) client_negotiation.cc:240] Negotiation successful
0619 16:43:41.209578 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12022,"mutex_wait_us":46,"thread_start_us":12148,"threads_started":1}
I20260619 16:43:41.210608 4964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:41.189559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47318 (local address 127.0.97.1:45727)
0619 16:43:41.200955 (+ 11396us) server_negotiation.cc:206] Beginning negotiation
0619 16:43:41.200964 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 16:43:41.201799 (+ 835us) server_negotiation.cc:407] Connection header received
0619 16:43:41.201919 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:43:41.201928 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:43:41.202212 (+ 284us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:43:41.202403 (+ 191us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:43:41.203342 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:41.204968 (+ 1626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:41.207296 (+ 2328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:41.208670 (+ 1374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:41.208783 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:43:41.208990 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0619 16:43:41.210169 (+ 1179us) server_negotiation.cc:299] Negotiation successful
0619 16:43:41.210323 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11211,"thread_start_us":11403,"threads_started":1}
I20260619 16:43:41.213295 4752 tablet_service.cc:1467] Processing CreateTablet for tablet 39a257029da0493c82aa71a6ab8cfe03 (DEFAULT_TABLE table=test-table [id=c20c8b68c4ca419f94f276483d642962]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:43:41.214761 4752 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 39a257029da0493c82aa71a6ab8cfe03. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:43:41.215081 4965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:41.190890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:34555 (local address 127.0.0.1:45088)
0619 16:43:41.202202 (+ 11312us) negotiation.cc:107] Waiting for socket to connect
0619 16:43:41.202217 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 16:43:41.202326 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:43:41.206559 (+ 4233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:43:41.206564 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:43:41.206590 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:43:41.207062 (+ 472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:43:41.207091 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:43:41.209280 (+ 2189us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:43:41.209286 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:43:41.212570 (+ 3284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:43:41.212584 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:43:41.214346 (+ 1762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:43:41.214353 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:43:41.214527 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:43:41.214563 (+ 36us) client_negotiation.cc:769] Sending connection context
0619 16:43:41.214661 (+ 98us) client_negotiation.cc:240] Negotiation successful
0619 16:43:41.214797 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11202,"thread_start_us":11057,"threads_started":1}
I20260619 16:43:41.215425 4966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:41.191584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45088 (local address 127.0.97.3:34555)
0619 16:43:41.204860 (+ 13276us) server_negotiation.cc:206] Beginning negotiation
0619 16:43:41.204871 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 16:43:41.204905 (+ 34us) server_negotiation.cc:407] Connection header received
0619 16:43:41.204998 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:43:41.205006 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:43:41.205241 (+ 235us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:43:41.207287 (+ 2046us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:43:41.207566 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:41.209112 (+ 1546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:41.212782 (+ 3670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:41.214135 (+ 1353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:41.214271 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:43:41.214496 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0619 16:43:41.214971 (+ 475us) server_negotiation.cc:299] Negotiation successful
0619 16:43:41.215127 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13088,"thread_start_us":14174,"threads_started":1}
I20260619 16:43:41.214967 4619 tablet_service.cc:1467] Processing CreateTablet for tablet 39a257029da0493c82aa71a6ab8cfe03 (DEFAULT_TABLE table=test-table [id=c20c8b68c4ca419f94f276483d642962]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:43:41.216218 4619 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 39a257029da0493c82aa71a6ab8cfe03. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:43:41.218089 4884 tablet_service.cc:1467] Processing CreateTablet for tablet 39a257029da0493c82aa71a6ab8cfe03 (DEFAULT_TABLE table=test-table [id=c20c8b68c4ca419f94f276483d642962]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:43:41.219285 4884 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 39a257029da0493c82aa71a6ab8cfe03. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:43:41.229200 4968 tablet_bootstrap.cc:492] T 39a257029da0493c82aa71a6ab8cfe03 P 42601d81443840cb88ea5a2c2ba6a1d5: Bootstrap starting.
I20260619 16:43:41.230700 4684 heartbeater.cc:503] Master 127.0.97.62:32779 was elected leader, sending a full tablet report...
I20260619 16:43:41.231052 4967 tablet_bootstrap.cc:492] T 39a257029da0493c82aa71a6ab8cfe03 P 9fe332844aef4d7badc14dea7b7266b9: Bootstrap starting.
I20260619 16:43:41.232669 4968 tablet_bootstrap.cc:654] T 39a257029da0493c82aa71a6ab8cfe03 P 42601d81443840cb88ea5a2c2ba6a1d5: Neither blocks nor log segments found. Creating new log.
I20260619 16:43:41.233897 4968 log.cc:824] T 39a257029da0493c82aa71a6ab8cfe03 P 42601d81443840cb88ea5a2c2ba6a1d5: Log is configured to *not* fsync() on all Append() calls
I20260619 16:43:41.234180 4967 tablet_bootstrap.cc:654] T 39a257029da0493c82aa71a6ab8cfe03 P 9fe332844aef4d7badc14dea7b7266b9: Neither blocks nor log segments found. Creating new log.
I20260619 16:43:41.235523 4967 log.cc:824] T 39a257029da0493c82aa71a6ab8cfe03 P 9fe332844aef4d7badc14dea7b7266b9: Log is configured to *not* fsync() on all Append() calls
I20260619 16:43:41.236477 4969 tablet_bootstrap.cc:492] T 39a257029da0493c82aa71a6ab8cfe03 P 57251729b4cb4277ac2cbe7a678f72df: Bootstrap starting.
I20260619 16:43:41.239692 4969 tablet_bootstrap.cc:654] T 39a257029da0493c82aa71a6ab8cfe03 P 57251729b4cb4277ac2cbe7a678f72df: Neither blocks nor log segments found. Creating new log.
I20260619 16:43:41.239732 4967 tablet_bootstrap.cc:492] T 39a257029da0493c82aa71a6ab8cfe03 P 9fe332844aef4d7badc14dea7b7266b9: No bootstrap required, opened a new log
I20260619 16:43:41.239820 4968 tablet_bootstrap.cc:492] T 39a257029da0493c82aa71a6ab8cfe03 P 42601d81443840cb88ea5a2c2ba6a1d5: No bootstrap required, opened a new log
I20260619 16:43:41.239960 4967 ts_tablet_manager.cc:1397] T 39a257029da0493c82aa71a6ab8cfe03 P 9fe332844aef4d7badc14dea7b7266b9: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260619 16:43:41.240065 4968 ts_tablet_manager.cc:1397] T 39a257029da0493c82aa71a6ab8cfe03 P 42601d81443840cb88ea5a2c2ba6a1d5: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.004s
I20260619 16:43:41.240782 4969 log.cc:824] T 39a257029da0493c82aa71a6ab8cfe03 P 57251729b4cb4277ac2cbe7a678f72df: Log is configured to *not* fsync() on all Append() calls
I20260619 16:43:41.243527 4969 tablet_bootstrap.cc:492] T 39a257029da0493c82aa71a6ab8cfe03 P 57251729b4cb4277ac2cbe7a678f72df: No bootstrap required, opened a new log
I20260619 16:43:41.243770 4969 ts_tablet_manager.cc:1397] T 39a257029da0493c82aa71a6ab8cfe03 P 57251729b4cb4277ac2cbe7a678f72df: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.005s
I20260619 16:43:41.246949 4967 raft_consensus.cc:348] T 39a257029da0493c82aa71a6ab8cfe03 P 9fe332844aef4d7badc14dea7b7266b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9fe332844aef4d7badc14dea7b7266b9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39019 } } peers { permanent_uuid: "57251729b4cb4277ac2cbe7a678f72df" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45727 } } peers { permanent_uuid: "42601d81443840cb88ea5a2c2ba6a1d5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34555 } }
I20260619 16:43:41.247269 4967 raft_consensus.cc:374] T 39a257029da0493c82aa71a6ab8cfe03 P 9fe332844aef4d7badc14dea7b7266b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:43:41.247371 4967 raft_consensus.cc:729] T 39a257029da0493c82aa71a6ab8cfe03 P 9fe332844aef4d7badc14dea7b7266b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9fe332844aef4d7badc14dea7b7266b9, State: Initialized, Role: FOLLOWER
I20260619 16:43:41.247900 4967 consensus_queue.cc:260] T 39a257029da0493c82aa71a6ab8cfe03 P 9fe332844aef4d7badc14dea7b7266b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9fe332844aef4d7badc14dea7b7266b9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39019 } } peers { permanent_uuid: "57251729b4cb4277ac2cbe7a678f72df" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45727 } } peers { permanent_uuid: "42601d81443840cb88ea5a2c2ba6a1d5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34555 } }
I20260619 16:43:41.248847 4817 heartbeater.cc:503] Master 127.0.97.62:32779 was elected leader, sending a full tablet report...
I20260619 16:43:41.249333 4967 ts_tablet_manager.cc:1428] T 39a257029da0493c82aa71a6ab8cfe03 P 9fe332844aef4d7badc14dea7b7266b9: Time spent starting tablet: real 0.009s user 0.010s sys 0.000s
I20260619 16:43:41.249578 4968 raft_consensus.cc:348] T 39a257029da0493c82aa71a6ab8cfe03 P 42601d81443840cb88ea5a2c2ba6a1d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9fe332844aef4d7badc14dea7b7266b9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39019 } } peers { permanent_uuid: "57251729b4cb4277ac2cbe7a678f72df" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45727 } } peers { permanent_uuid: "42601d81443840cb88ea5a2c2ba6a1d5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34555 } }
I20260619 16:43:41.249987 4968 raft_consensus.cc:374] T 39a257029da0493c82aa71a6ab8cfe03 P 42601d81443840cb88ea5a2c2ba6a1d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:43:41.250079 4968 raft_consensus.cc:729] T 39a257029da0493c82aa71a6ab8cfe03 P 42601d81443840cb88ea5a2c2ba6a1d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 42601d81443840cb88ea5a2c2ba6a1d5, State: Initialized, Role: FOLLOWER
I20260619 16:43:41.250627 4968 consensus_queue.cc:260] T 39a257029da0493c82aa71a6ab8cfe03 P 42601d81443840cb88ea5a2c2ba6a1d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9fe332844aef4d7badc14dea7b7266b9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39019 } } peers { permanent_uuid: "57251729b4cb4277ac2cbe7a678f72df" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45727 } } peers { permanent_uuid: "42601d81443840cb88ea5a2c2ba6a1d5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34555 } }
I20260619 16:43:41.252521 4949 heartbeater.cc:503] Master 127.0.97.62:32779 was elected leader, sending a full tablet report...
I20260619 16:43:41.253286 4968 ts_tablet_manager.cc:1428] T 39a257029da0493c82aa71a6ab8cfe03 P 42601d81443840cb88ea5a2c2ba6a1d5: Time spent starting tablet: real 0.013s user 0.014s sys 0.000s
I20260619 16:43:41.254814 4969 raft_consensus.cc:348] T 39a257029da0493c82aa71a6ab8cfe03 P 57251729b4cb4277ac2cbe7a678f72df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9fe332844aef4d7badc14dea7b7266b9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39019 } } peers { permanent_uuid: "57251729b4cb4277ac2cbe7a678f72df" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45727 } } peers { permanent_uuid: "42601d81443840cb88ea5a2c2ba6a1d5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34555 } }
I20260619 16:43:41.255210 4969 raft_consensus.cc:374] T 39a257029da0493c82aa71a6ab8cfe03 P 57251729b4cb4277ac2cbe7a678f72df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:43:41.255306 4969 raft_consensus.cc:729] T 39a257029da0493c82aa71a6ab8cfe03 P 57251729b4cb4277ac2cbe7a678f72df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 57251729b4cb4277ac2cbe7a678f72df, State: Initialized, Role: FOLLOWER
I20260619 16:43:41.255849 4969 consensus_queue.cc:260] T 39a257029da0493c82aa71a6ab8cfe03 P 57251729b4cb4277ac2cbe7a678f72df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9fe332844aef4d7badc14dea7b7266b9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39019 } } peers { permanent_uuid: "57251729b4cb4277ac2cbe7a678f72df" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45727 } } peers { permanent_uuid: "42601d81443840cb88ea5a2c2ba6a1d5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34555 } }
I20260619 16:43:41.257182 4969 ts_tablet_manager.cc:1428] T 39a257029da0493c82aa71a6ab8cfe03 P 57251729b4cb4277ac2cbe7a678f72df: Time spent starting tablet: real 0.013s user 0.006s sys 0.007s
W20260619 16:43:41.371302 4950 tablet.cc:2369] T 39a257029da0493c82aa71a6ab8cfe03 P 42601d81443840cb88ea5a2c2ba6a1d5: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 16:43:41.406152 4818 tablet.cc:2369] T 39a257029da0493c82aa71a6ab8cfe03 P 9fe332844aef4d7badc14dea7b7266b9: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 16:43:41.455749 4685 tablet.cc:2369] T 39a257029da0493c82aa71a6ab8cfe03 P 57251729b4cb4277ac2cbe7a678f72df: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 16:43:41.530002 4973 raft_consensus.cc:482] T 39a257029da0493c82aa71a6ab8cfe03 P 9fe332844aef4d7badc14dea7b7266b9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 16:43:41.530274 4973 raft_consensus.cc:504] T 39a257029da0493c82aa71a6ab8cfe03 P 9fe332844aef4d7badc14dea7b7266b9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9fe332844aef4d7badc14dea7b7266b9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39019 } } peers { permanent_uuid: "57251729b4cb4277ac2cbe7a678f72df" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45727 } } peers { permanent_uuid: "42601d81443840cb88ea5a2c2ba6a1d5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34555 } }
I20260619 16:43:41.531865 4973 leader_election.cc:288] T 39a257029da0493c82aa71a6ab8cfe03 P 9fe332844aef4d7badc14dea7b7266b9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 57251729b4cb4277ac2cbe7a678f72df (127.0.97.1:45727), 42601d81443840cb88ea5a2c2ba6a1d5 (127.0.97.3:34555)
I20260619 16:43:41.538014 4964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:41.532274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:36721 (local address 127.0.97.1:45727)
0619 16:43:41.532471 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0619 16:43:41.532477 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:43:41.533112 (+ 635us) server_negotiation.cc:407] Connection header received
0619 16:43:41.533306 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:43:41.533310 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:43:41.533365 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:43:41.533438 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:43:41.533970 (+ 532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:41.534733 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:41.536417 (+ 1684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:41.537144 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:41.537272 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:43:41.537369 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0619 16:43:41.537629 (+ 260us) server_negotiation.cc:299] Negotiation successful
0619 16:43:41.537790 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260619 16:43:41.538147 4966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:41.532275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:54783 (local address 127.0.97.3:34555)
0619 16:43:41.532470 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0619 16:43:41.532478 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 16:43:41.533268 (+ 790us) server_negotiation.cc:407] Connection header received
0619 16:43:41.533439 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:43:41.533443 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:43:41.533496 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:43:41.533564 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:43:41.534175 (+ 611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:41.534886 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:41.536553 (+ 1667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:41.537311 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:41.537419 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:43:41.537510 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0619 16:43:41.537803 (+ 293us) server_negotiation.cc:299] Negotiation successful
0619 16:43:41.537919 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260619 16:43:41.538173 4976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:41.532296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45727 (local address 127.0.97.2:36721)
0619 16:43:41.533004 (+ 708us) negotiation.cc:107] Waiting for socket to connect
0619 16:43:41.533019 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 16:43:41.533158 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:43:41.533467 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:43:41.533471 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:43:41.533493 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:43:41.533839 (+ 346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:43:41.533847 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:43:41.534856 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:43:41.534862 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:43:41.536265 (+ 1403us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:43:41.536283 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:43:41.537356 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:43:41.537362 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:43:41.537490 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:43:41.537512 (+ 22us) client_negotiation.cc:769] Sending connection context
0619 16:43:41.537597 (+ 85us) client_negotiation.cc:240] Negotiation successful
0619 16:43:41.537721 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":599,"mutex_wait_us":3,"spinlock_wait_cycles":16640,"thread_start_us":349,"threads_started":1}
I20260619 16:43:41.538293 4977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:41.532412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:34555 (local address 127.0.97.2:54783)
0619 16:43:41.533165 (+ 753us) negotiation.cc:107] Waiting for socket to connect
0619 16:43:41.533179 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 16:43:41.533304 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:43:41.533602 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:43:41.533610 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:43:41.533657 (+ 47us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:43:41.534041 (+ 384us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:43:41.534049 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:43:41.535023 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:43:41.535028 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:43:41.536415 (+ 1387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:43:41.536425 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:43:41.537503 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:43:41.537509 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:43:41.537650 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:43:41.537675 (+ 25us) client_negotiation.cc:769] Sending connection context
0619 16:43:41.537777 (+ 102us) client_negotiation.cc:240] Negotiation successful
0619 16:43:41.537895 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":653,"mutex_wait_us":1,"spinlock_wait_cycles":768,"thread_start_us":469,"threads_started":1}
I20260619 16:43:41.539108 4904 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39a257029da0493c82aa71a6ab8cfe03" candidate_uuid: "9fe332844aef4d7badc14dea7b7266b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42601d81443840cb88ea5a2c2ba6a1d5" is_pre_election: true
I20260619 16:43:41.539112 4639 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39a257029da0493c82aa71a6ab8cfe03" candidate_uuid: "9fe332844aef4d7badc14dea7b7266b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "57251729b4cb4277ac2cbe7a678f72df" is_pre_election: true
I20260619 16:43:41.539534 4904 raft_consensus.cc:2454] T 39a257029da0493c82aa71a6ab8cfe03 P 42601d81443840cb88ea5a2c2ba6a1d5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9fe332844aef4d7badc14dea7b7266b9 in term 0.
I20260619 16:43:41.539534 4639 raft_consensus.cc:2454] T 39a257029da0493c82aa71a6ab8cfe03 P 57251729b4cb4277ac2cbe7a678f72df [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9fe332844aef4d7badc14dea7b7266b9 in term 0.
I20260619 16:43:41.540115 4708 leader_election.cc:302] T 39a257029da0493c82aa71a6ab8cfe03 P 9fe332844aef4d7badc14dea7b7266b9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42601d81443840cb88ea5a2c2ba6a1d5, 9fe332844aef4d7badc14dea7b7266b9; no voters:
I20260619 16:43:41.540445 4973 raft_consensus.cc:2781] T 39a257029da0493c82aa71a6ab8cfe03 P 9fe332844aef4d7badc14dea7b7266b9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 16:43:41.540584 4973 raft_consensus.cc:482] T 39a257029da0493c82aa71a6ab8cfe03 P 9fe332844aef4d7badc14dea7b7266b9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 16:43:41.540676 4973 raft_consensus.cc:3037] T 39a257029da0493c82aa71a6ab8cfe03 P 9fe332844aef4d7badc14dea7b7266b9 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:43:41.542717 4973 raft_consensus.cc:504] T 39a257029da0493c82aa71a6ab8cfe03 P 9fe332844aef4d7badc14dea7b7266b9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9fe332844aef4d7badc14dea7b7266b9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39019 } } peers { permanent_uuid: "57251729b4cb4277ac2cbe7a678f72df" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45727 } } peers { permanent_uuid: "42601d81443840cb88ea5a2c2ba6a1d5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34555 } }
I20260619 16:43:41.543223 4973 leader_election.cc:288] T 39a257029da0493c82aa71a6ab8cfe03 P 9fe332844aef4d7badc14dea7b7266b9 [CANDIDATE]: Term 1 election: Requested vote from peers 57251729b4cb4277ac2cbe7a678f72df (127.0.97.1:45727), 42601d81443840cb88ea5a2c2ba6a1d5 (127.0.97.3:34555)
I20260619 16:43:41.543589 4639 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39a257029da0493c82aa71a6ab8cfe03" candidate_uuid: "9fe332844aef4d7badc14dea7b7266b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "57251729b4cb4277ac2cbe7a678f72df"
I20260619 16:43:41.543639 4904 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39a257029da0493c82aa71a6ab8cfe03" candidate_uuid: "9fe332844aef4d7badc14dea7b7266b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42601d81443840cb88ea5a2c2ba6a1d5"
I20260619 16:43:41.543738 4639 raft_consensus.cc:3037] T 39a257029da0493c82aa71a6ab8cfe03 P 57251729b4cb4277ac2cbe7a678f72df [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:43:41.543841 4904 raft_consensus.cc:3037] T 39a257029da0493c82aa71a6ab8cfe03 P 42601d81443840cb88ea5a2c2ba6a1d5 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:43:41.545861 4639 raft_consensus.cc:2454] T 39a257029da0493c82aa71a6ab8cfe03 P 57251729b4cb4277ac2cbe7a678f72df [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9fe332844aef4d7badc14dea7b7266b9 in term 1.
I20260619 16:43:41.545949 4904 raft_consensus.cc:2454] T 39a257029da0493c82aa71a6ab8cfe03 P 42601d81443840cb88ea5a2c2ba6a1d5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9fe332844aef4d7badc14dea7b7266b9 in term 1.
I20260619 16:43:41.546247 4706 leader_election.cc:302] T 39a257029da0493c82aa71a6ab8cfe03 P 9fe332844aef4d7badc14dea7b7266b9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 57251729b4cb4277ac2cbe7a678f72df, 9fe332844aef4d7badc14dea7b7266b9; no voters:
I20260619 16:43:41.546507 4973 raft_consensus.cc:2781] T 39a257029da0493c82aa71a6ab8cfe03 P 9fe332844aef4d7badc14dea7b7266b9 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:43:41.546975 4973 raft_consensus.cc:686] T 39a257029da0493c82aa71a6ab8cfe03 P 9fe332844aef4d7badc14dea7b7266b9 [term 1 LEADER]: Becoming Leader. State: Replica: 9fe332844aef4d7badc14dea7b7266b9, State: Running, Role: LEADER
I20260619 16:43:41.547421 4973 consensus_queue.cc:237] T 39a257029da0493c82aa71a6ab8cfe03 P 9fe332844aef4d7badc14dea7b7266b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9fe332844aef4d7badc14dea7b7266b9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39019 } } peers { permanent_uuid: "57251729b4cb4277ac2cbe7a678f72df" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45727 } } peers { permanent_uuid: "42601d81443840cb88ea5a2c2ba6a1d5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34555 } }
I20260619 16:43:41.551592 4496 catalog_manager.cc:5495] T 39a257029da0493c82aa71a6ab8cfe03 P 9fe332844aef4d7badc14dea7b7266b9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9fe332844aef4d7badc14dea7b7266b9 (127.0.97.2). New cstate: current_term: 1 leader_uuid: "9fe332844aef4d7badc14dea7b7266b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9fe332844aef4d7badc14dea7b7266b9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39019 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "57251729b4cb4277ac2cbe7a678f72df" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45727 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "42601d81443840cb88ea5a2c2ba6a1d5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34555 } health_report { overall_health: UNKNOWN } } }
I20260619 16:43:41.652287 4962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:41.646657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53286 (local address 127.0.97.2:39019)
0619 16:43:41.646827 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0619 16:43:41.646831 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:43:41.646848 (+ 17us) server_negotiation.cc:407] Connection header received
0619 16:43:41.646958 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:43:41.646962 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:43:41.647042 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:43:41.647158 (+ 116us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 16:43:41.647514 (+ 356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:41.648258 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:41.650180 (+ 1922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:41.650607 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:41.650703 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:43:41.651213 (+ 510us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 16:43:41.651773 (+ 560us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 16:43:41.651874 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0619 16:43:41.651968 (+ 94us) server_negotiation.cc:299] Negotiation successful
0619 16:43:41.652030 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260619 16:43:41.661331 4904 raft_consensus.cc:1261] T 39a257029da0493c82aa71a6ab8cfe03 P 42601d81443840cb88ea5a2c2ba6a1d5 [term 1 FOLLOWER]: Refusing update from remote peer 9fe332844aef4d7badc14dea7b7266b9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:43:41.661597 4639 raft_consensus.cc:1261] T 39a257029da0493c82aa71a6ab8cfe03 P 57251729b4cb4277ac2cbe7a678f72df [term 1 FOLLOWER]: Refusing update from remote peer 9fe332844aef4d7badc14dea7b7266b9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:43:41.662357 4978 consensus_queue.cc:1035] T 39a257029da0493c82aa71a6ab8cfe03 P 9fe332844aef4d7badc14dea7b7266b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "42601d81443840cb88ea5a2c2ba6a1d5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34555 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:43:41.662730 4973 consensus_queue.cc:1035] T 39a257029da0493c82aa71a6ab8cfe03 P 9fe332844aef4d7badc14dea7b7266b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "57251729b4cb4277ac2cbe7a678f72df" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45727 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:43:41.672926 4981 mvcc.cc:205] Tried to move back new op lower bound from 7298610879115419648 to 7298610878661685248. Current Snapshot: MvccSnapshot[applied={T|T < 7298610879115419648}]
I20260619 16:43:41.688097 4495 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56114:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:43:41.688340 4495 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56114:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:43:41.691823 4495 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f993ca2899fd4e779ab2d78a4697b890: Sending DeleteTablet for 3 replicas of tablet 39a257029da0493c82aa71a6ab8cfe03
I20260619 16:43:41.692941 4752 tablet_service.cc:1514] Processing DeleteTablet for tablet 39a257029da0493c82aa71a6ab8cfe03 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:43:41 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:53280
I20260619 16:43:41.692948 4619 tablet_service.cc:1514] Processing DeleteTablet for tablet 39a257029da0493c82aa71a6ab8cfe03 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:43:41 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:47318
I20260619 16:43:41.693141 4884 tablet_service.cc:1514] Processing DeleteTablet for tablet 39a257029da0493c82aa71a6ab8cfe03 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:43:41 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:45088
I20260619 16:43:41.695322 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu pid 4557
I20260619 16:43:41.705432 4993 tablet_replica.cc:330] T 39a257029da0493c82aa71a6ab8cfe03 P 9fe332844aef4d7badc14dea7b7266b9: stopping tablet replica
I20260619 16:43:41.706223 4994 tablet_replica.cc:330] T 39a257029da0493c82aa71a6ab8cfe03 P 42601d81443840cb88ea5a2c2ba6a1d5: stopping tablet replica
I20260619 16:43:41.706678 4994 raft_consensus.cc:2229] T 39a257029da0493c82aa71a6ab8cfe03 P 42601d81443840cb88ea5a2c2ba6a1d5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 16:43:41.707584 4994 raft_consensus.cc:2258] T 39a257029da0493c82aa71a6ab8cfe03 P 42601d81443840cb88ea5a2c2ba6a1d5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:43:41.707069 4993 raft_consensus.cc:2229] T 39a257029da0493c82aa71a6ab8cfe03 P 9fe332844aef4d7badc14dea7b7266b9 [term 1 LEADER]: Raft consensus shutting down.
I20260619 16:43:41.709352 4993 raft_consensus.cc:2258] T 39a257029da0493c82aa71a6ab8cfe03 P 9fe332844aef4d7badc14dea7b7266b9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:43:41.710330 4994 ts_tablet_manager.cc:1905] T 39a257029da0493c82aa71a6ab8cfe03 P 42601d81443840cb88ea5a2c2ba6a1d5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:43:41.711134 4993 ts_tablet_manager.cc:1905] T 39a257029da0493c82aa71a6ab8cfe03 P 9fe332844aef4d7badc14dea7b7266b9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:43:41.717806 4994 ts_tablet_manager.cc:1918] T 39a257029da0493c82aa71a6ab8cfe03 P 42601d81443840cb88ea5a2c2ba6a1d5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 16:43:41.718101 4993 ts_tablet_manager.cc:1918] T 39a257029da0493c82aa71a6ab8cfe03 P 9fe332844aef4d7badc14dea7b7266b9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 16:43:41.718349 4993 log.cc:1196] T 39a257029da0493c82aa71a6ab8cfe03 P 9fe332844aef4d7badc14dea7b7266b9: Deleting WAL directory at /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887385903558-388-0/minicluster-data/ts-1/wal/wals/39a257029da0493c82aa71a6ab8cfe03
I20260619 16:43:41.718967 4993 ts_tablet_manager.cc:1939] T 39a257029da0493c82aa71a6ab8cfe03 P 9fe332844aef4d7badc14dea7b7266b9: Deleting consensus metadata
I20260619 16:43:41.719700 4994 log.cc:1196] T 39a257029da0493c82aa71a6ab8cfe03 P 42601d81443840cb88ea5a2c2ba6a1d5: Deleting WAL directory at /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887385903558-388-0/minicluster-data/ts-2/wal/wals/39a257029da0493c82aa71a6ab8cfe03
I20260619 16:43:41.719839 4992 tablet_replica.cc:330] T 39a257029da0493c82aa71a6ab8cfe03 P 57251729b4cb4277ac2cbe7a678f72df: stopping tablet replica
I20260619 16:43:41.720203 4994 ts_tablet_manager.cc:1939] T 39a257029da0493c82aa71a6ab8cfe03 P 42601d81443840cb88ea5a2c2ba6a1d5: Deleting consensus metadata
I20260619 16:43:41.720331 4992 raft_consensus.cc:2229] T 39a257029da0493c82aa71a6ab8cfe03 P 57251729b4cb4277ac2cbe7a678f72df [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 16:43:41.720779 4482 catalog_manager.cc:4841] TS 9fe332844aef4d7badc14dea7b7266b9 (127.0.97.2:39019): tablet 39a257029da0493c82aa71a6ab8cfe03 (table test-table [id=c20c8b68c4ca419f94f276483d642962]) successfully deleted
I20260619 16:43:41.720858 4992 raft_consensus.cc:2258] T 39a257029da0493c82aa71a6ab8cfe03 P 57251729b4cb4277ac2cbe7a678f72df [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:43:41.721794 4483 catalog_manager.cc:4841] TS 42601d81443840cb88ea5a2c2ba6a1d5 (127.0.97.3:34555): tablet 39a257029da0493c82aa71a6ab8cfe03 (table test-table [id=c20c8b68c4ca419f94f276483d642962]) successfully deleted
I20260619 16:43:41.723515 4992 ts_tablet_manager.cc:1905] T 39a257029da0493c82aa71a6ab8cfe03 P 57251729b4cb4277ac2cbe7a678f72df: Deleting tablet data with delete state TABLET_DATA_DELETED
Jun 19 16:43:41 dist-test-slave-wvz7 krb5kdc[4452](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887419, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260619 16:43:41.730367 4992 ts_tablet_manager.cc:1918] T 39a257029da0493c82aa71a6ab8cfe03 P 57251729b4cb4277ac2cbe7a678f72df: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 16:43:41.730568 4992 log.cc:1196] T 39a257029da0493c82aa71a6ab8cfe03 P 57251729b4cb4277ac2cbe7a678f72df: Deleting WAL directory at /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887385903558-388-0/minicluster-data/ts-0/wal/wals/39a257029da0493c82aa71a6ab8cfe03
I20260619 16:43:41.731012 4964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:41.696959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47334 (local address 127.0.97.1:45727)
0619 16:43:41.697147 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0619 16:43:41.697154 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:43:41.712612 (+ 15458us) server_negotiation.cc:407] Connection header received
0619 16:43:41.713092 (+ 480us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:43:41.713099 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:43:41.713207 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:43:41.713282 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:43:41.715066 (+ 1784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:41.719258 (+ 4192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:41.720913 (+ 1655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:41.721518 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:41.721602 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:43:41.725575 (+ 3973us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:43:41.725589 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:43:41.725606 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:43:41.725664 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:43:41.728386 (+ 2722us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:43:41.729001 (+ 615us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:43:41.729008 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:43:41.729010 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:43:41.729140 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:43:41.729482 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:43:41.729486 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:43:41.729488 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:43:41.729858 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:43:41.730121 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0619 16:43:41.730423 (+ 302us) server_negotiation.cc:299] Negotiation successful
0619 16:43:41.730640 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260619 16:43:41.731096 4992 ts_tablet_manager.cc:1939] T 39a257029da0493c82aa71a6ab8cfe03 P 57251729b4cb4277ac2cbe7a678f72df: Deleting consensus metadata
I20260619 16:43:41.731639 4679 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:43:41.818498 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu with pid 4557
I20260619 16:43:41.818662 4481 catalog_manager.cc:4841] TS 57251729b4cb4277ac2cbe7a678f72df (127.0.97.1:45727): tablet 39a257029da0493c82aa71a6ab8cfe03 (table test-table [id=c20c8b68c4ca419f94f276483d642962]) successfully deleted
I20260619 16:43:41.837723 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu pid 4690
Jun 19 16:43:41 dist-test-slave-wvz7 krb5kdc[4452](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887419, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260619 16:43:41.853431 4962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:41.838463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53300 (local address 127.0.97.2:39019)
0619 16:43:41.838625 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0619 16:43:41.838629 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:43:41.838699 (+ 70us) server_negotiation.cc:407] Connection header received
0619 16:43:41.838867 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:43:41.838870 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:43:41.838935 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:43:41.839019 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:43:41.840506 (+ 1487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:41.841299 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:41.843190 (+ 1891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:41.843588 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:41.843695 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:43:41.848268 (+ 4573us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:43:41.848281 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:43:41.848295 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:43:41.848325 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:43:41.851018 (+ 2693us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:43:41.851576 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:43:41.851582 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:43:41.851585 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:43:41.851713 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:43:41.852036 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:43:41.852040 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:43:41.852042 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:43:41.852384 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:43:41.852685 (+ 301us) server_negotiation.cc:1035] Waiting for connection context
0619 16:43:41.852913 (+ 228us) server_negotiation.cc:299] Negotiation successful
0619 16:43:41.853116 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260619 16:43:41.854192 4812 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:43:41.964839 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu with pid 4690
I20260619 16:43:41.978266 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu pid 4822
Jun 19 16:43:41 dist-test-slave-wvz7 krb5kdc[4452](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887419, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260619 16:43:41.991806 4966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:41.978982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45094 (local address 127.0.97.3:34555)
0619 16:43:41.979147 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0619 16:43:41.979152 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:43:41.979198 (+ 46us) server_negotiation.cc:407] Connection header received
0619 16:43:41.979374 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:43:41.979378 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:43:41.979453 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:43:41.979533 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:43:41.980831 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:41.981538 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:41.982729 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:41.983129 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:41.983229 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:43:41.987097 (+ 3868us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:43:41.987112 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:43:41.987127 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:43:41.987161 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:43:41.989507 (+ 2346us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:43:41.990175 (+ 668us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:43:41.990182 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:43:41.990185 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:43:41.990309 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:43:41.990576 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:43:41.990579 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:43:41.990581 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:43:41.990903 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:43:41.991095 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0619 16:43:41.991338 (+ 243us) server_negotiation.cc:299] Negotiation successful
0619 16:43:41.991488 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260619 16:43:41.992378 4944 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:43:42.077338 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu with pid 4822
I20260619 16:43:42.089805 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu pid 4465
I20260619 16:43:42.090865 4526 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:43:42.194666 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu with pid 4465
2026-06-19T16:43:42Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (4131 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-06-19T16:43:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T16:43:42Z Disabled control of system clock
I20260619 16:43:42.250855 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu
/tmp/dist-test-taskEs2EH8/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781887385903558-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781887385903558-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781887385903558-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781887385903558-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37991
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:44649
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:37991
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260619 16:43:42.550743 5003 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:43:42.551026 5003 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:43:42.551106 5003 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:43:42.558100 5003 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:43:42.558203 5003 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:43:42.558238 5003 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:43:42.558290 5003 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:43:42.558331 5003 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:43:42.567248 5003 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44649
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781887385903558-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781887385903558-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:37991
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37991
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781887385903558-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781887385903558-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:43:42.568424 5003 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:43:42.570152 5003 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:43:42.581081 5011 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:43:42.581066 5009 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:43:42.581115 5008 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:43:42.581825 5003 server_base.cc:1034] running on GCE node
I20260619 16:43:42.582531 5003 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:43:42.583863 5003 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:43:42.585043 5003 hybrid_clock.cc:648] HybridClock initialized: now 1781887422584992 us; error 45 us; skew 500 ppm
I20260619 16:43:42.585366 5003 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 16:43:42.587411 5003 webserver.cc:460] Webserver started at http://127.0.97.62:36605/ using document root <none> and password file <none>
I20260619 16:43:42.588033 5003 fs_manager.cc:359] Metadata directory not provided
I20260619 16:43:42.588135 5003 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:43:42.588429 5003 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:43:42.591173 5003 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781887385903558-388-0/minicluster-data/master-0/data/instance:
uuid: "3e562a5903ca4065972de95aa01bbb47"
format_stamp: "Formatted at 2026-06-19 16:43:42 on dist-test-slave-wvz7"
server_key: "f32449ab66f2c203e8d396d5dd900bea"
server_key_iv: "229839747f890dfd19352fd05168a647"
server_key_version: "encryptionkey@0"
I20260619 16:43:42.591917 5003 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781887385903558-388-0/minicluster-data/master-0/wal/instance:
uuid: "3e562a5903ca4065972de95aa01bbb47"
format_stamp: "Formatted at 2026-06-19 16:43:42 on dist-test-slave-wvz7"
server_key: "f32449ab66f2c203e8d396d5dd900bea"
server_key_iv: "229839747f890dfd19352fd05168a647"
server_key_version: "encryptionkey@0"
I20260619 16:43:42.596936 5003 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260619 16:43:42.600260 5017 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:43:42.601637 5003 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 16:43:42.601835 5003 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781887385903558-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781887385903558-388-0/minicluster-data/master-0/wal
uuid: "3e562a5903ca4065972de95aa01bbb47"
format_stamp: "Formatted at 2026-06-19 16:43:42 on dist-test-slave-wvz7"
server_key: "f32449ab66f2c203e8d396d5dd900bea"
server_key_iv: "229839747f890dfd19352fd05168a647"
server_key_version: "encryptionkey@0"
I20260619 16:43:42.602022 5003 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781887385903558-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781887385903558-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781887385903558-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:43:42.607311 5003 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:43:42.608022 5003 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:43:42.608281 5003 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:43:42.624222 5003 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:37991
I20260619 16:43:42.624240 5068 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:37991 every 8 connection(s)
I20260619 16:43:42.625500 5003 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781887385903558-388-0/minicluster-data/master-0/data/info.pb
I20260619 16:43:42.625938 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu as pid 5003
I20260619 16:43:42.626113 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781887385903558-388-0/minicluster-data/master-0/wal/instance
I20260619 16:43:42.626935 388 external_mini_cluster.cc:1428] Setting key d90e63814cd8e829c2f9bcfff7ba21c0
I20260619 16:43:42.630082 5069 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:43:42.634931 5072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:42.628993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54710 (local address 127.0.97.62:37991)
0619 16:43:42.630025 (+ 1032us) server_negotiation.cc:206] Beginning negotiation
0619 16:43:42.630036 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 16:43:42.630079 (+ 43us) server_negotiation.cc:407] Connection header received
0619 16:43:42.631608 (+ 1529us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:43:42.631630 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:43:42.632011 (+ 381us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:43:42.632377 (+ 366us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:43:42.633323 (+ 946us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:43:42.633333 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:43:42.633345 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:43:42.633359 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:43:42.633429 (+ 70us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:43:42.633444 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:43:42.633654 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0619 16:43:42.633808 (+ 154us) server_negotiation.cc:299] Negotiation successful
0619 16:43:42.634032 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":548,"thread_start_us":323,"threads_started":1}
I20260619 16:43:42.640542 5069 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3e562a5903ca4065972de95aa01bbb47: Bootstrap starting.
I20260619 16:43:42.643280 5069 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3e562a5903ca4065972de95aa01bbb47: Neither blocks nor log segments found. Creating new log.
I20260619 16:43:42.644258 5069 log.cc:824] T 00000000000000000000000000000000 P 3e562a5903ca4065972de95aa01bbb47: Log is configured to *not* fsync() on all Append() calls
I20260619 16:43:42.646718 5069 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3e562a5903ca4065972de95aa01bbb47: No bootstrap required, opened a new log
I20260619 16:43:42.653333 5069 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3e562a5903ca4065972de95aa01bbb47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e562a5903ca4065972de95aa01bbb47" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37991 } }
I20260619 16:43:42.653638 5069 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3e562a5903ca4065972de95aa01bbb47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:43:42.653728 5069 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3e562a5903ca4065972de95aa01bbb47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e562a5903ca4065972de95aa01bbb47, State: Initialized, Role: FOLLOWER
I20260619 16:43:42.654309 5069 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3e562a5903ca4065972de95aa01bbb47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e562a5903ca4065972de95aa01bbb47" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37991 } }
I20260619 16:43:42.654508 5069 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3e562a5903ca4065972de95aa01bbb47 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:43:42.654618 5069 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3e562a5903ca4065972de95aa01bbb47 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:43:42.654776 5069 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3e562a5903ca4065972de95aa01bbb47 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:43:42.656734 5069 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3e562a5903ca4065972de95aa01bbb47 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e562a5903ca4065972de95aa01bbb47" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37991 } }
I20260619 16:43:42.657207 5069 leader_election.cc:302] T 00000000000000000000000000000000 P 3e562a5903ca4065972de95aa01bbb47 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3e562a5903ca4065972de95aa01bbb47; no voters:
I20260619 16:43:42.657815 5069 leader_election.cc:288] T 00000000000000000000000000000000 P 3e562a5903ca4065972de95aa01bbb47 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 16:43:42.657929 5074 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3e562a5903ca4065972de95aa01bbb47 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:43:42.658541 5074 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3e562a5903ca4065972de95aa01bbb47 [term 1 LEADER]: Becoming Leader. State: Replica: 3e562a5903ca4065972de95aa01bbb47, State: Running, Role: LEADER
I20260619 16:43:42.659073 5074 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3e562a5903ca4065972de95aa01bbb47 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e562a5903ca4065972de95aa01bbb47" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37991 } }
I20260619 16:43:42.659637 5069 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3e562a5903ca4065972de95aa01bbb47 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:43:42.661571 5075 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e562a5903ca4065972de95aa01bbb47 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3e562a5903ca4065972de95aa01bbb47" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e562a5903ca4065972de95aa01bbb47" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37991 } } }
I20260619 16:43:42.661967 5075 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e562a5903ca4065972de95aa01bbb47 [sys.catalog]: This master's current role is: LEADER
I20260619 16:43:42.661808 5076 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e562a5903ca4065972de95aa01bbb47 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3e562a5903ca4065972de95aa01bbb47. Latest consensus state: current_term: 1 leader_uuid: "3e562a5903ca4065972de95aa01bbb47" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e562a5903ca4065972de95aa01bbb47" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37991 } } }
I20260619 16:43:42.662091 5076 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e562a5903ca4065972de95aa01bbb47 [sys.catalog]: This master's current role is: LEADER
I20260619 16:43:42.663359 5083 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:43:42.668013 5083 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:43:42.676319 5083 catalog_manager.cc:1347] Generated new cluster ID: 72982174085046d08dd7dc67929edecd
I20260619 16:43:42.676483 5083 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:43:42.692963 5083 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 16:43:42.693326 5083 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:43:42.704582 5083 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3e562a5903ca4065972de95aa01bbb47: Generated new TSK 0
I20260619 16:43:42.705394 5083 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 16:43:42.715488 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu
/tmp/dist-test-taskEs2EH8/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781887385903558-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781887385903558-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781887385903558-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781887385903558-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37991
--builtin_ntp_servers=127.0.97.20:44649
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260619 16:43:43.044198 5093 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:43:43.044476 5093 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:43:43.044562 5093 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:43:43.051659 5093 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:43:43.051759 5093 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:43:43.051867 5093 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260619 16:43:43.060681 5093 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44649
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781887385903558-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781887385903558-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781887385903558-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37991
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781887385903558-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:43:43.061981 5093 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:43:43.063768 5093 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:43:43.075652 5098 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:43:43.075680 5101 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:43:43.075877 5099 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:43:43.076809 5093 server_base.cc:1034] running on GCE node
I20260619 16:43:43.077355 5093 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:43:43.078218 5093 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:43:43.079407 5093 hybrid_clock.cc:648] HybridClock initialized: now 1781887423079365 us; error 32 us; skew 500 ppm
I20260619 16:43:43.079694 5093 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 16:43:43.081679 5093 webserver.cc:460] Webserver started at http://127.0.97.1:35099/ using document root <none> and password file <none>
I20260619 16:43:43.082314 5093 fs_manager.cc:359] Metadata directory not provided
I20260619 16:43:43.082401 5093 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:43:43.082695 5093 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:43:43.085353 5093 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781887385903558-388-0/minicluster-data/ts-0/data/instance:
uuid: "590f0821b945497ba33007ad423a60b9"
format_stamp: "Formatted at 2026-06-19 16:43:43 on dist-test-slave-wvz7"
server_key: "a046205ef05ede99583440551594e35c"
server_key_iv: "529eba51b1e91d1e5df7730bc96f1151"
server_key_version: "encryptionkey@0"
I20260619 16:43:43.086100 5093 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781887385903558-388-0/minicluster-data/ts-0/wal/instance:
uuid: "590f0821b945497ba33007ad423a60b9"
format_stamp: "Formatted at 2026-06-19 16:43:43 on dist-test-slave-wvz7"
server_key: "a046205ef05ede99583440551594e35c"
server_key_iv: "529eba51b1e91d1e5df7730bc96f1151"
server_key_version: "encryptionkey@0"
I20260619 16:43:43.090711 5093 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260619 16:43:43.093729 5107 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:43:43.095068 5093 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260619 16:43:43.095225 5093 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781887385903558-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781887385903558-388-0/minicluster-data/ts-0/wal
uuid: "590f0821b945497ba33007ad423a60b9"
format_stamp: "Formatted at 2026-06-19 16:43:43 on dist-test-slave-wvz7"
server_key: "a046205ef05ede99583440551594e35c"
server_key_iv: "529eba51b1e91d1e5df7730bc96f1151"
server_key_version: "encryptionkey@0"
I20260619 16:43:43.095441 5093 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781887385903558-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781887385903558-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781887385903558-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:43:43.101264 5093 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:43:43.102123 5093 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:43:43.102389 5093 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:43:43.103245 5093 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:43:43.104660 5093 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:43:43.104766 5093 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:43:43.104884 5093 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:43:43.104959 5093 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:43:43.134428 5093 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:37023
I20260619 16:43:43.134467 5219 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:37023 every 8 connection(s)
I20260619 16:43:43.135826 5093 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781887385903558-388-0/minicluster-data/ts-0/data/info.pb
I20260619 16:43:43.143682 5222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:43.138154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:40437 (local address 127.0.97.62:37991)
0619 16:43:43.138570 (+ 416us) server_negotiation.cc:206] Beginning negotiation
0619 16:43:43.138575 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:43:43.139731 (+ 1156us) server_negotiation.cc:407] Connection header received
0619 16:43:43.141450 (+ 1719us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:43:43.141457 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:43:43.141532 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:43:43.141635 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:43:43.142935 (+ 1300us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:43:43.142942 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:43:43.142947 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:43:43.142969 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:43:43.143020 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:43:43.143035 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:43:43.143166 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0619 16:43:43.143308 (+ 142us) server_negotiation.cc:299] Negotiation successful
0619 16:43:43.143369 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":311,"thread_start_us":217,"threads_started":1}
I20260619 16:43:43.144627 5223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:43.138640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37991 (local address 127.0.97.1:40437)
0619 16:43:43.139570 (+ 930us) negotiation.cc:107] Waiting for socket to connect
0619 16:43:43.139617 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0619 16:43:43.141189 (+ 1572us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:43:43.141908 (+ 719us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:43:43.141916 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:43:43.142020 (+ 104us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:43:43.142635 (+ 615us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 16:43:43.142637 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:43:43.142673 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 16:43:43.142759 (+ 86us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:43:43.143130 (+ 371us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:43:43.143136 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:43:43.143141 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 16:43:43.143354 (+ 213us) client_negotiation.cc:240] Negotiation successful
0619 16:43:43.143581 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":439,"thread_start_us":247,"threads_started":1}
I20260619 16:43:43.144832 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu as pid 5093
I20260619 16:43:43.145061 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781887385903558-388-0/minicluster-data/ts-0/wal/instance
I20260619 16:43:43.145902 388 external_mini_cluster.cc:1428] Setting key 8a6c0a74da74f4b3721e6a7f3fbec976
I20260619 16:43:43.146296 5220 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37991
I20260619 16:43:43.146601 5220 heartbeater.cc:464] Registering TS with master...
I20260619 16:43:43.147188 5220 heartbeater.cc:511] Master 127.0.97.62:37991 requested a full tablet report, sending...
I20260619 16:43:43.148814 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu
/tmp/dist-test-taskEs2EH8/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781887385903558-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781887385903558-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781887385903558-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781887385903558-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37991
--builtin_ntp_servers=127.0.97.20:44649
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260619 16:43:43.149467 5034 ts_manager.cc:194] Registered new tserver with Master: 590f0821b945497ba33007ad423a60b9 (127.0.97.1:37023)
W20260619 16:43:43.452363 5224 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:43:43.452631 5224 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:43:43.452759 5224 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:43:43.459779 5224 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:43:43.459882 5224 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:43:43.460024 5224 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260619 16:43:43.468827 5224 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44649
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781887385903558-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781887385903558-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781887385903558-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37991
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781887385903558-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:43:43.470016 5224 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:43:43.471702 5224 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:43:43.483417 5229 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:43:43.483436 5230 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:43:43.484122 5232 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:43:43.484102 5224 server_base.cc:1034] running on GCE node
I20260619 16:43:43.484687 5224 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:43:43.485569 5224 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:43:43.486783 5224 hybrid_clock.cc:648] HybridClock initialized: now 1781887423486720 us; error 49 us; skew 500 ppm
I20260619 16:43:43.487082 5224 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 16:43:43.489224 5224 webserver.cc:460] Webserver started at http://127.0.97.2:37075/ using document root <none> and password file <none>
I20260619 16:43:43.489921 5224 fs_manager.cc:359] Metadata directory not provided
I20260619 16:43:43.490023 5224 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:43:43.490314 5224 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:43:43.493044 5224 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781887385903558-388-0/minicluster-data/ts-1/data/instance:
uuid: "14daf912eac14e86bd37f419caa15787"
format_stamp: "Formatted at 2026-06-19 16:43:43 on dist-test-slave-wvz7"
server_key: "6bcd894f50c1faf3b346dc512b2a5316"
server_key_iv: "921dcafd3a482326f2b3750853c19878"
server_key_version: "encryptionkey@0"
I20260619 16:43:43.493811 5224 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781887385903558-388-0/minicluster-data/ts-1/wal/instance:
uuid: "14daf912eac14e86bd37f419caa15787"
format_stamp: "Formatted at 2026-06-19 16:43:43 on dist-test-slave-wvz7"
server_key: "6bcd894f50c1faf3b346dc512b2a5316"
server_key_iv: "921dcafd3a482326f2b3750853c19878"
server_key_version: "encryptionkey@0"
I20260619 16:43:43.498632 5224 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260619 16:43:43.501878 5238 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:43:43.503123 5224 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260619 16:43:43.503243 5224 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781887385903558-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781887385903558-388-0/minicluster-data/ts-1/wal
uuid: "14daf912eac14e86bd37f419caa15787"
format_stamp: "Formatted at 2026-06-19 16:43:43 on dist-test-slave-wvz7"
server_key: "6bcd894f50c1faf3b346dc512b2a5316"
server_key_iv: "921dcafd3a482326f2b3750853c19878"
server_key_version: "encryptionkey@0"
I20260619 16:43:43.503412 5224 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781887385903558-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781887385903558-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781887385903558-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:43:43.508517 5224 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:43:43.509183 5224 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:43:43.509387 5224 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:43:43.510205 5224 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:43:43.511431 5224 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:43:43.511508 5224 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:43:43.511593 5224 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:43:43.511646 5224 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:43:43.540326 5224 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:44647
I20260619 16:43:43.540370 5350 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:44647 every 8 connection(s)
I20260619 16:43:43.541540 5224 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781887385903558-388-0/minicluster-data/ts-1/data/info.pb
I20260619 16:43:43.545292 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu as pid 5224
I20260619 16:43:43.545447 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781887385903558-388-0/minicluster-data/ts-1/wal/instance
I20260619 16:43:43.546000 388 external_mini_cluster.cc:1428] Setting key 41e7a3657aebd0d9996cf67b0100793c
I20260619 16:43:43.548787 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu
/tmp/dist-test-taskEs2EH8/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781887385903558-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781887385903558-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781887385903558-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781887385903558-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37991
--builtin_ntp_servers=127.0.97.20:44649
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260619 16:43:43.549965 5222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:43.543877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:49955 (local address 127.0.97.62:37991)
0619 16:43:43.544063 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0619 16:43:43.544070 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:43:43.545673 (+ 1603us) server_negotiation.cc:407] Connection header received
0619 16:43:43.547618 (+ 1945us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:43:43.547623 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:43:43.547678 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:43:43.547771 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:43:43.549243 (+ 1472us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:43:43.549247 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:43:43.549250 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:43:43.549266 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:43:43.549303 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:43:43.549318 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:43:43.549431 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0619 16:43:43.549622 (+ 191us) server_negotiation.cc:299] Negotiation successful
0619 16:43:43.549679 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260619 16:43:43.551108 5353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:43.544406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37991 (local address 127.0.97.2:49955)
0619 16:43:43.545463 (+ 1057us) negotiation.cc:107] Waiting for socket to connect
0619 16:43:43.545510 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0619 16:43:43.547366 (+ 1856us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:43:43.548003 (+ 637us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:43:43.548015 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:43:43.548138 (+ 123us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:43:43.548795 (+ 657us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 16:43:43.548798 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:43:43.548847 (+ 49us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 16:43:43.549072 (+ 225us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:43:43.549414 (+ 342us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:43:43.549422 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:43:43.549431 (+ 9us) client_negotiation.cc:769] Sending connection context
0619 16:43:43.549700 (+ 269us) client_negotiation.cc:240] Negotiation successful
0619 16:43:43.549988 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":474,"thread_start_us":278,"threads_started":1}
I20260619 16:43:43.552368 5351 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37991
I20260619 16:43:43.552630 5351 heartbeater.cc:464] Registering TS with master...
I20260619 16:43:43.553169 5351 heartbeater.cc:511] Master 127.0.97.62:37991 requested a full tablet report, sending...
I20260619 16:43:43.554419 5034 ts_manager.cc:194] Registered new tserver with Master: 14daf912eac14e86bd37f419caa15787 (127.0.97.2:44647)
W20260619 16:43:43.851076 5354 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:43:43.851330 5354 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:43:43.851454 5354 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:43:43.858458 5354 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:43:43.858551 5354 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:43:43.858691 5354 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260619 16:43:43.867726 5354 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44649
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781887385903558-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781887385903558-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781887385903558-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37991
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781887385903558-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:43:43.868894 5354 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:43:43.870589 5354 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:43:43.882815 5359 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:43:43.882793 5360 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:43:43.882865 5362 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:43:43.883687 5354 server_base.cc:1034] running on GCE node
I20260619 16:43:43.884254 5354 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:43:43.885404 5354 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:43:43.886629 5354 hybrid_clock.cc:648] HybridClock initialized: now 1781887423886561 us; error 54 us; skew 500 ppm
I20260619 16:43:43.886989 5354 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 16:43:43.890219 5354 webserver.cc:460] Webserver started at http://127.0.97.3:36843/ using document root <none> and password file <none>
I20260619 16:43:43.891019 5354 fs_manager.cc:359] Metadata directory not provided
I20260619 16:43:43.891161 5354 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:43:43.891528 5354 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:43:43.894871 5354 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781887385903558-388-0/minicluster-data/ts-2/data/instance:
uuid: "c2e859f3cb0d46f197282eafbd14fe0f"
format_stamp: "Formatted at 2026-06-19 16:43:43 on dist-test-slave-wvz7"
server_key: "b34b111b4fc02087f4b7f0352d8a998b"
server_key_iv: "8946047c1285378b2b7059691426266f"
server_key_version: "encryptionkey@0"
I20260619 16:43:43.895594 5354 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781887385903558-388-0/minicluster-data/ts-2/wal/instance:
uuid: "c2e859f3cb0d46f197282eafbd14fe0f"
format_stamp: "Formatted at 2026-06-19 16:43:43 on dist-test-slave-wvz7"
server_key: "b34b111b4fc02087f4b7f0352d8a998b"
server_key_iv: "8946047c1285378b2b7059691426266f"
server_key_version: "encryptionkey@0"
I20260619 16:43:43.900487 5354 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260619 16:43:43.903697 5368 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:43:43.905004 5354 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 16:43:43.905130 5354 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781887385903558-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781887385903558-388-0/minicluster-data/ts-2/wal
uuid: "c2e859f3cb0d46f197282eafbd14fe0f"
format_stamp: "Formatted at 2026-06-19 16:43:43 on dist-test-slave-wvz7"
server_key: "b34b111b4fc02087f4b7f0352d8a998b"
server_key_iv: "8946047c1285378b2b7059691426266f"
server_key_version: "encryptionkey@0"
I20260619 16:43:43.905309 5354 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781887385903558-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781887385903558-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781887385903558-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:43:43.910835 5354 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:43:43.911547 5354 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:43:43.911783 5354 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:43:43.912567 5354 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:43:43.913890 5354 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:43:43.913969 5354 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:43:43.914045 5354 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:43:43.914096 5354 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:43:43.941649 5354 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:41581
I20260619 16:43:43.941720 5480 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:41581 every 8 connection(s)
I20260619 16:43:43.942910 5354 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781887385903558-388-0/minicluster-data/ts-2/data/info.pb
I20260619 16:43:43.945933 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu as pid 5354
I20260619 16:43:43.946067 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781887385903558-388-0/minicluster-data/ts-2/wal/instance
I20260619 16:43:43.946555 388 external_mini_cluster.cc:1428] Setting key 99613b3165ea0aadde9dda1f07a0b3a1
I20260619 16:43:43.950985 5222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:43.945251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:57211 (local address 127.0.97.62:37991)
0619 16:43:43.945439 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0619 16:43:43.945446 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:43:43.947070 (+ 1624us) server_negotiation.cc:407] Connection header received
0619 16:43:43.949110 (+ 2040us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:43:43.949115 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:43:43.949175 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:43:43.949252 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:43:43.950430 (+ 1178us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:43:43.950434 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:43:43.950437 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:43:43.950451 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:43:43.950480 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:43:43.950491 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:43:43.950580 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0619 16:43:43.950747 (+ 167us) server_negotiation.cc:299] Negotiation successful
0619 16:43:43.950791 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260619 16:43:43.951870 5483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:43.945874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37991 (local address 127.0.97.3:57211)
0619 16:43:43.946896 (+ 1022us) negotiation.cc:107] Waiting for socket to connect
0619 16:43:43.946940 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0619 16:43:43.948841 (+ 1901us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:43:43.949456 (+ 615us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:43:43.949465 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:43:43.949555 (+ 90us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:43:43.950174 (+ 619us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 16:43:43.950177 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:43:43.950213 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 16:43:43.950294 (+ 81us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:43:43.950587 (+ 293us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:43:43.950593 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:43:43.950600 (+ 7us) client_negotiation.cc:769] Sending connection context
0619 16:43:43.950799 (+ 199us) client_negotiation.cc:240] Negotiation successful
0619 16:43:43.950988 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":486,"thread_start_us":315,"threads_started":1}
I20260619 16:43:43.953014 5481 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37991
I20260619 16:43:43.953261 5481 heartbeater.cc:464] Registering TS with master...
I20260619 16:43:43.953778 5481 heartbeater.cc:511] Master 127.0.97.62:37991 requested a full tablet report, sending...
I20260619 16:43:43.954988 5034 ts_manager.cc:194] Registered new tserver with Master: c2e859f3cb0d46f197282eafbd14fe0f (127.0.97.3:41581)
I20260619 16:43:43.961397 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 16:43:43.967772 5222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:43.965529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43330 (local address 127.0.97.62:37991)
0619 16:43:43.965706 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0619 16:43:43.965713 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:43:43.966157 (+ 444us) server_negotiation.cc:407] Connection header received
0619 16:43:43.966300 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:43:43.966303 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:43:43.966350 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:43:43.966424 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:43:43.967262 (+ 838us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:43:43.967267 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:43:43.967270 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:43:43.967284 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:43:43.967314 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:43:43.967324 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:43:43.967415 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0619 16:43:43.967466 (+ 51us) server_negotiation.cc:299] Negotiation successful
0619 16:43:43.967506 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260619 16:43:43.971359 5034 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43330:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 16:43:43.977713 5034 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 16:43:44.003840 5493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:43.988590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:44647 (local address 127.0.0.1:42554)
0619 16:43:43.999511 (+ 10921us) negotiation.cc:107] Waiting for socket to connect
0619 16:43:43.999555 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0619 16:43:43.999714 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:43:44.000445 (+ 731us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:43:44.000453 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:43:44.000561 (+ 108us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:43:44.001436 (+ 875us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 16:43:44.001440 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:43:44.001473 (+ 33us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 16:43:44.001566 (+ 93us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:43:44.003073 (+ 1507us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:43:44.003086 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:43:44.003095 (+ 9us) client_negotiation.cc:769] Sending connection context
0619 16:43:44.003261 (+ 166us) client_negotiation.cc:240] Negotiation successful
0619 16:43:44.003354 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10688,"thread_start_us":10503,"threads_started":1}
I20260619 16:43:44.005126 5494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:43.988721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42554 (local address 127.0.97.2:44647)
0619 16:43:43.999097 (+ 10376us) server_negotiation.cc:206] Beginning negotiation
0619 16:43:43.999108 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 16:43:43.999834 (+ 726us) server_negotiation.cc:407] Connection header received
0619 16:43:43.999911 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:43:43.999919 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:43:44.000230 (+ 311us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:43:44.001756 (+ 1526us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:43:44.002816 (+ 1060us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:43:44.002826 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:43:44.002842 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:43:44.002864 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:43:44.002928 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:43:44.002948 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:43:44.004591 (+ 1643us) server_negotiation.cc:1035] Waiting for connection context
0619 16:43:44.004679 (+ 88us) server_negotiation.cc:299] Negotiation successful
0619 16:43:44.004783 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10170,"thread_start_us":9998,"threads_started":1}
I20260619 16:43:44.006986 5496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:43.988947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52820 (local address 127.0.97.1:37023)
0619 16:43:44.002721 (+ 13774us) server_negotiation.cc:206] Beginning negotiation
0619 16:43:44.002729 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 16:43:44.002756 (+ 27us) server_negotiation.cc:407] Connection header received
0619 16:43:44.002816 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:43:44.002824 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:43:44.003067 (+ 243us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:43:44.003231 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:43:44.006262 (+ 3031us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:43:44.006271 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:43:44.006285 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:43:44.006313 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:43:44.006370 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:43:44.006388 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:43:44.006513 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0619 16:43:44.006617 (+ 104us) server_negotiation.cc:299] Negotiation successful
0619 16:43:44.006707 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13642,"thread_start_us":13531,"threads_started":1}
I20260619 16:43:44.007409 5495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:43.988594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:37023 (local address 127.0.0.1:52820)
0619 16:43:44.001050 (+ 12456us) negotiation.cc:107] Waiting for socket to connect
0619 16:43:44.001067 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 16:43:44.001170 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:43:44.004154 (+ 2984us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:43:44.004160 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:43:44.004185 (+ 25us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:43:44.006043 (+ 1858us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 16:43:44.006047 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:43:44.006061 (+ 14us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 16:43:44.006103 (+ 42us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:43:44.006490 (+ 387us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:43:44.006495 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:43:44.006501 (+ 6us) client_negotiation.cc:769] Sending connection context
0619 16:43:44.006634 (+ 133us) client_negotiation.cc:240] Negotiation successful
0619 16:43:44.006695 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12325,"mutex_wait_us":194,"thread_start_us":12435,"threads_started":1}
I20260619 16:43:44.009212 5493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:44.002804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:41581 (local address 127.0.0.1:53944)
0619 16:43:44.004296 (+ 1492us) negotiation.cc:107] Waiting for socket to connect
0619 16:43:44.004314 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 16:43:44.004391 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:43:44.006756 (+ 2365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:43:44.006760 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:43:44.006796 (+ 36us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:43:44.007279 (+ 483us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 16:43:44.007282 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:43:44.007301 (+ 19us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 16:43:44.007342 (+ 41us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:43:44.008847 (+ 1505us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:43:44.008851 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:43:44.008856 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 16:43:44.008950 (+ 94us) client_negotiation.cc:240] Negotiation successful
0619 16:43:44.008994 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1390,"thread_start_us":3149,"threads_started":1}
I20260619 16:43:44.009415 5497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:44.000301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53944 (local address 127.0.97.3:41581)
0619 16:43:44.006201 (+ 5900us) server_negotiation.cc:206] Beginning negotiation
0619 16:43:44.006209 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 16:43:44.006241 (+ 32us) server_negotiation.cc:407] Connection header received
0619 16:43:44.006305 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:43:44.006312 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:43:44.006596 (+ 284us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:43:44.007080 (+ 484us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:43:44.008573 (+ 1493us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:43:44.008582 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:43:44.008596 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:43:44.008615 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:43:44.008668 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:43:44.008721 (+ 53us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:43:44.008911 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0619 16:43:44.008968 (+ 57us) server_negotiation.cc:299] Negotiation successful
0619 16:43:44.009049 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5736,"thread_start_us":5581,"threads_started":1}
I20260619 16:43:44.010085 5286 tablet_service.cc:1467] Processing CreateTablet for tablet 203c077ee85d481bb4ed7d9d496a2c47 (DEFAULT_TABLE table=test-table [id=e04bd8d791104896b636564129874fc1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:43:44.011432 5286 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 203c077ee85d481bb4ed7d9d496a2c47. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:43:44.011278 5155 tablet_service.cc:1467] Processing CreateTablet for tablet 203c077ee85d481bb4ed7d9d496a2c47 (DEFAULT_TABLE table=test-table [id=e04bd8d791104896b636564129874fc1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:43:44.012470 5155 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 203c077ee85d481bb4ed7d9d496a2c47. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:43:44.012969 5416 tablet_service.cc:1467] Processing CreateTablet for tablet 203c077ee85d481bb4ed7d9d496a2c47 (DEFAULT_TABLE table=test-table [id=e04bd8d791104896b636564129874fc1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:43:44.014252 5416 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 203c077ee85d481bb4ed7d9d496a2c47. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:43:44.023938 5499 tablet_bootstrap.cc:492] T 203c077ee85d481bb4ed7d9d496a2c47 P 590f0821b945497ba33007ad423a60b9: Bootstrap starting.
I20260619 16:43:44.025027 5500 tablet_bootstrap.cc:492] T 203c077ee85d481bb4ed7d9d496a2c47 P 14daf912eac14e86bd37f419caa15787: Bootstrap starting.
I20260619 16:43:44.027275 5499 tablet_bootstrap.cc:654] T 203c077ee85d481bb4ed7d9d496a2c47 P 590f0821b945497ba33007ad423a60b9: Neither blocks nor log segments found. Creating new log.
I20260619 16:43:44.028306 5500 tablet_bootstrap.cc:654] T 203c077ee85d481bb4ed7d9d496a2c47 P 14daf912eac14e86bd37f419caa15787: Neither blocks nor log segments found. Creating new log.
I20260619 16:43:44.028445 5499 log.cc:824] T 203c077ee85d481bb4ed7d9d496a2c47 P 590f0821b945497ba33007ad423a60b9: Log is configured to *not* fsync() on all Append() calls
I20260619 16:43:44.029004 5501 tablet_bootstrap.cc:492] T 203c077ee85d481bb4ed7d9d496a2c47 P c2e859f3cb0d46f197282eafbd14fe0f: Bootstrap starting.
I20260619 16:43:44.029860 5500 log.cc:824] T 203c077ee85d481bb4ed7d9d496a2c47 P 14daf912eac14e86bd37f419caa15787: Log is configured to *not* fsync() on all Append() calls
I20260619 16:43:44.031760 5501 tablet_bootstrap.cc:654] T 203c077ee85d481bb4ed7d9d496a2c47 P c2e859f3cb0d46f197282eafbd14fe0f: Neither blocks nor log segments found. Creating new log.
I20260619 16:43:44.032570 5501 log.cc:824] T 203c077ee85d481bb4ed7d9d496a2c47 P c2e859f3cb0d46f197282eafbd14fe0f: Log is configured to *not* fsync() on all Append() calls
I20260619 16:43:44.035131 5501 tablet_bootstrap.cc:492] T 203c077ee85d481bb4ed7d9d496a2c47 P c2e859f3cb0d46f197282eafbd14fe0f: No bootstrap required, opened a new log
I20260619 16:43:44.035370 5501 ts_tablet_manager.cc:1397] T 203c077ee85d481bb4ed7d9d496a2c47 P c2e859f3cb0d46f197282eafbd14fe0f: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260619 16:43:44.035427 5499 tablet_bootstrap.cc:492] T 203c077ee85d481bb4ed7d9d496a2c47 P 590f0821b945497ba33007ad423a60b9: No bootstrap required, opened a new log
I20260619 16:43:44.035427 5500 tablet_bootstrap.cc:492] T 203c077ee85d481bb4ed7d9d496a2c47 P 14daf912eac14e86bd37f419caa15787: No bootstrap required, opened a new log
I20260619 16:43:44.035665 5500 ts_tablet_manager.cc:1397] T 203c077ee85d481bb4ed7d9d496a2c47 P 14daf912eac14e86bd37f419caa15787: Time spent bootstrapping tablet: real 0.011s user 0.008s sys 0.000s
I20260619 16:43:44.035676 5499 ts_tablet_manager.cc:1397] T 203c077ee85d481bb4ed7d9d496a2c47 P 590f0821b945497ba33007ad423a60b9: Time spent bootstrapping tablet: real 0.013s user 0.005s sys 0.006s
I20260619 16:43:44.042311 5501 raft_consensus.cc:348] T 203c077ee85d481bb4ed7d9d496a2c47 P c2e859f3cb0d46f197282eafbd14fe0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14daf912eac14e86bd37f419caa15787" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44647 } } peers { permanent_uuid: "590f0821b945497ba33007ad423a60b9" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37023 } } peers { permanent_uuid: "c2e859f3cb0d46f197282eafbd14fe0f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41581 } }
I20260619 16:43:44.042642 5501 raft_consensus.cc:374] T 203c077ee85d481bb4ed7d9d496a2c47 P c2e859f3cb0d46f197282eafbd14fe0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:43:44.042749 5501 raft_consensus.cc:729] T 203c077ee85d481bb4ed7d9d496a2c47 P c2e859f3cb0d46f197282eafbd14fe0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2e859f3cb0d46f197282eafbd14fe0f, State: Initialized, Role: FOLLOWER
I20260619 16:43:44.043347 5501 consensus_queue.cc:260] T 203c077ee85d481bb4ed7d9d496a2c47 P c2e859f3cb0d46f197282eafbd14fe0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14daf912eac14e86bd37f419caa15787" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44647 } } peers { permanent_uuid: "590f0821b945497ba33007ad423a60b9" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37023 } } peers { permanent_uuid: "c2e859f3cb0d46f197282eafbd14fe0f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41581 } }
I20260619 16:43:44.044330 5481 heartbeater.cc:503] Master 127.0.97.62:37991 was elected leader, sending a full tablet report...
I20260619 16:43:44.044988 5501 ts_tablet_manager.cc:1428] T 203c077ee85d481bb4ed7d9d496a2c47 P c2e859f3cb0d46f197282eafbd14fe0f: Time spent starting tablet: real 0.010s user 0.010s sys 0.000s
I20260619 16:43:44.045485 5499 raft_consensus.cc:348] T 203c077ee85d481bb4ed7d9d496a2c47 P 590f0821b945497ba33007ad423a60b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14daf912eac14e86bd37f419caa15787" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44647 } } peers { permanent_uuid: "590f0821b945497ba33007ad423a60b9" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37023 } } peers { permanent_uuid: "c2e859f3cb0d46f197282eafbd14fe0f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41581 } }
I20260619 16:43:44.045485 5500 raft_consensus.cc:348] T 203c077ee85d481bb4ed7d9d496a2c47 P 14daf912eac14e86bd37f419caa15787 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14daf912eac14e86bd37f419caa15787" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44647 } } peers { permanent_uuid: "590f0821b945497ba33007ad423a60b9" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37023 } } peers { permanent_uuid: "c2e859f3cb0d46f197282eafbd14fe0f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41581 } }
I20260619 16:43:44.045928 5500 raft_consensus.cc:374] T 203c077ee85d481bb4ed7d9d496a2c47 P 14daf912eac14e86bd37f419caa15787 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:43:44.045953 5499 raft_consensus.cc:374] T 203c077ee85d481bb4ed7d9d496a2c47 P 590f0821b945497ba33007ad423a60b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:43:44.046027 5499 raft_consensus.cc:729] T 203c077ee85d481bb4ed7d9d496a2c47 P 590f0821b945497ba33007ad423a60b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 590f0821b945497ba33007ad423a60b9, State: Initialized, Role: FOLLOWER
I20260619 16:43:44.046027 5500 raft_consensus.cc:729] T 203c077ee85d481bb4ed7d9d496a2c47 P 14daf912eac14e86bd37f419caa15787 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14daf912eac14e86bd37f419caa15787, State: Initialized, Role: FOLLOWER
I20260619 16:43:44.046614 5500 consensus_queue.cc:260] T 203c077ee85d481bb4ed7d9d496a2c47 P 14daf912eac14e86bd37f419caa15787 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14daf912eac14e86bd37f419caa15787" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44647 } } peers { permanent_uuid: "590f0821b945497ba33007ad423a60b9" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37023 } } peers { permanent_uuid: "c2e859f3cb0d46f197282eafbd14fe0f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41581 } }
I20260619 16:43:44.046614 5499 consensus_queue.cc:260] T 203c077ee85d481bb4ed7d9d496a2c47 P 590f0821b945497ba33007ad423a60b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14daf912eac14e86bd37f419caa15787" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44647 } } peers { permanent_uuid: "590f0821b945497ba33007ad423a60b9" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37023 } } peers { permanent_uuid: "c2e859f3cb0d46f197282eafbd14fe0f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41581 } }
I20260619 16:43:44.048203 5220 heartbeater.cc:503] Master 127.0.97.62:37991 was elected leader, sending a full tablet report...
I20260619 16:43:44.048257 5499 ts_tablet_manager.cc:1428] T 203c077ee85d481bb4ed7d9d496a2c47 P 590f0821b945497ba33007ad423a60b9: Time spent starting tablet: real 0.012s user 0.004s sys 0.008s
I20260619 16:43:44.048516 5500 ts_tablet_manager.cc:1428] T 203c077ee85d481bb4ed7d9d496a2c47 P 14daf912eac14e86bd37f419caa15787: Time spent starting tablet: real 0.013s user 0.009s sys 0.004s
I20260619 16:43:44.049842 5351 heartbeater.cc:503] Master 127.0.97.62:37991 was elected leader, sending a full tablet report...
I20260619 16:43:44.100342 5506 raft_consensus.cc:482] T 203c077ee85d481bb4ed7d9d496a2c47 P 14daf912eac14e86bd37f419caa15787 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 16:43:44.100630 5506 raft_consensus.cc:504] T 203c077ee85d481bb4ed7d9d496a2c47 P 14daf912eac14e86bd37f419caa15787 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14daf912eac14e86bd37f419caa15787" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44647 } } peers { permanent_uuid: "590f0821b945497ba33007ad423a60b9" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37023 } } peers { permanent_uuid: "c2e859f3cb0d46f197282eafbd14fe0f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41581 } }
I20260619 16:43:44.102149 5506 leader_election.cc:288] T 203c077ee85d481bb4ed7d9d496a2c47 P 14daf912eac14e86bd37f419caa15787 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 590f0821b945497ba33007ad423a60b9 (127.0.97.1:37023), c2e859f3cb0d46f197282eafbd14fe0f (127.0.97.3:41581)
I20260619 16:43:44.105139 5497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:44.102518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:37509 (local address 127.0.97.3:41581)
0619 16:43:44.102722 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0619 16:43:44.102729 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:43:44.103440 (+ 711us) server_negotiation.cc:407] Connection header received
0619 16:43:44.103597 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:43:44.103602 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:43:44.103661 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:43:44.103732 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:43:44.104578 (+ 846us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:43:44.104583 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:43:44.104587 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:43:44.104605 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:43:44.104639 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:43:44.104653 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:43:44.104760 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0619 16:43:44.104860 (+ 100us) server_negotiation.cc:299] Negotiation successful
0619 16:43:44.104915 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260619 16:43:44.105355 5496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:44.102510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:54127 (local address 127.0.97.1:37023)
0619 16:43:44.102714 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0619 16:43:44.102722 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 16:43:44.103461 (+ 739us) server_negotiation.cc:407] Connection header received
0619 16:43:44.103638 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:43:44.103643 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:43:44.103688 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:43:44.103772 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:43:44.104729 (+ 957us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:43:44.104734 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:43:44.104739 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:43:44.104756 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:43:44.104790 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:43:44.104807 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:43:44.104901 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0619 16:43:44.105068 (+ 167us) server_negotiation.cc:299] Negotiation successful
0619 16:43:44.105119 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260619 16:43:44.105522 5508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:44.102481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:37023 (local address 127.0.97.2:54127)
0619 16:43:44.103364 (+ 883us) negotiation.cc:107] Waiting for socket to connect
0619 16:43:44.103378 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 16:43:44.103525 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:43:44.103803 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:43:44.103807 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:43:44.103828 (+ 21us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:43:44.104543 (+ 715us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 16:43:44.104546 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:43:44.104560 (+ 14us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 16:43:44.104598 (+ 38us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:43:44.104945 (+ 347us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:43:44.104954 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:43:44.104958 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 16:43:44.105091 (+ 133us) client_negotiation.cc:240] Negotiation successful
0619 16:43:44.105136 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":792,"spinlock_wait_cycles":23424,"thread_start_us":416,"threads_started":1}
I20260619 16:43:44.105497 5509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:44.102440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:41581 (local address 127.0.97.2:37509)
0619 16:43:44.103331 (+ 891us) negotiation.cc:107] Waiting for socket to connect
0619 16:43:44.103346 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 16:43:44.103467 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:43:44.103775 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:43:44.103779 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:43:44.103800 (+ 21us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:43:44.104395 (+ 595us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 16:43:44.104397 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:43:44.104410 (+ 13us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 16:43:44.104444 (+ 34us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:43:44.104736 (+ 292us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:43:44.104741 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:43:44.104745 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 16:43:44.104866 (+ 121us) client_negotiation.cc:240] Negotiation successful
0619 16:43:44.104914 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":745,"thread_start_us":507,"threads_started":1}
I20260619 16:43:44.106600 5175 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "203c077ee85d481bb4ed7d9d496a2c47" candidate_uuid: "14daf912eac14e86bd37f419caa15787" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "590f0821b945497ba33007ad423a60b9" is_pre_election: true
I20260619 16:43:44.106604 5436 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "203c077ee85d481bb4ed7d9d496a2c47" candidate_uuid: "14daf912eac14e86bd37f419caa15787" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2e859f3cb0d46f197282eafbd14fe0f" is_pre_election: true
I20260619 16:43:44.107132 5436 raft_consensus.cc:2454] T 203c077ee85d481bb4ed7d9d496a2c47 P c2e859f3cb0d46f197282eafbd14fe0f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 14daf912eac14e86bd37f419caa15787 in term 0.
I20260619 16:43:44.107132 5175 raft_consensus.cc:2454] T 203c077ee85d481bb4ed7d9d496a2c47 P 590f0821b945497ba33007ad423a60b9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 14daf912eac14e86bd37f419caa15787 in term 0.
I20260619 16:43:44.107781 5239 leader_election.cc:302] T 203c077ee85d481bb4ed7d9d496a2c47 P 14daf912eac14e86bd37f419caa15787 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 14daf912eac14e86bd37f419caa15787, 590f0821b945497ba33007ad423a60b9, c2e859f3cb0d46f197282eafbd14fe0f; no voters:
I20260619 16:43:44.108170 5506 raft_consensus.cc:2781] T 203c077ee85d481bb4ed7d9d496a2c47 P 14daf912eac14e86bd37f419caa15787 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 16:43:44.108301 5506 raft_consensus.cc:482] T 203c077ee85d481bb4ed7d9d496a2c47 P 14daf912eac14e86bd37f419caa15787 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 16:43:44.108390 5506 raft_consensus.cc:3037] T 203c077ee85d481bb4ed7d9d496a2c47 P 14daf912eac14e86bd37f419caa15787 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:43:44.110666 5506 raft_consensus.cc:504] T 203c077ee85d481bb4ed7d9d496a2c47 P 14daf912eac14e86bd37f419caa15787 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14daf912eac14e86bd37f419caa15787" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44647 } } peers { permanent_uuid: "590f0821b945497ba33007ad423a60b9" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37023 } } peers { permanent_uuid: "c2e859f3cb0d46f197282eafbd14fe0f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41581 } }
I20260619 16:43:44.111173 5506 leader_election.cc:288] T 203c077ee85d481bb4ed7d9d496a2c47 P 14daf912eac14e86bd37f419caa15787 [CANDIDATE]: Term 1 election: Requested vote from peers 590f0821b945497ba33007ad423a60b9 (127.0.97.1:37023), c2e859f3cb0d46f197282eafbd14fe0f (127.0.97.3:41581)
I20260619 16:43:44.111629 5175 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "203c077ee85d481bb4ed7d9d496a2c47" candidate_uuid: "14daf912eac14e86bd37f419caa15787" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "590f0821b945497ba33007ad423a60b9"
I20260619 16:43:44.111629 5436 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "203c077ee85d481bb4ed7d9d496a2c47" candidate_uuid: "14daf912eac14e86bd37f419caa15787" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2e859f3cb0d46f197282eafbd14fe0f"
I20260619 16:43:44.111840 5436 raft_consensus.cc:3037] T 203c077ee85d481bb4ed7d9d496a2c47 P c2e859f3cb0d46f197282eafbd14fe0f [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:43:44.111840 5175 raft_consensus.cc:3037] T 203c077ee85d481bb4ed7d9d496a2c47 P 590f0821b945497ba33007ad423a60b9 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:43:44.115103 5436 raft_consensus.cc:2454] T 203c077ee85d481bb4ed7d9d496a2c47 P c2e859f3cb0d46f197282eafbd14fe0f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 14daf912eac14e86bd37f419caa15787 in term 1.
I20260619 16:43:44.115104 5175 raft_consensus.cc:2454] T 203c077ee85d481bb4ed7d9d496a2c47 P 590f0821b945497ba33007ad423a60b9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 14daf912eac14e86bd37f419caa15787 in term 1.
I20260619 16:43:44.115535 5240 leader_election.cc:302] T 203c077ee85d481bb4ed7d9d496a2c47 P 14daf912eac14e86bd37f419caa15787 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 14daf912eac14e86bd37f419caa15787, 590f0821b945497ba33007ad423a60b9, c2e859f3cb0d46f197282eafbd14fe0f; no voters:
I20260619 16:43:44.115878 5506 raft_consensus.cc:2781] T 203c077ee85d481bb4ed7d9d496a2c47 P 14daf912eac14e86bd37f419caa15787 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:43:44.116284 5506 raft_consensus.cc:686] T 203c077ee85d481bb4ed7d9d496a2c47 P 14daf912eac14e86bd37f419caa15787 [term 1 LEADER]: Becoming Leader. State: Replica: 14daf912eac14e86bd37f419caa15787, State: Running, Role: LEADER
I20260619 16:43:44.116698 5506 consensus_queue.cc:237] T 203c077ee85d481bb4ed7d9d496a2c47 P 14daf912eac14e86bd37f419caa15787 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14daf912eac14e86bd37f419caa15787" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44647 } } peers { permanent_uuid: "590f0821b945497ba33007ad423a60b9" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37023 } } peers { permanent_uuid: "c2e859f3cb0d46f197282eafbd14fe0f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41581 } }
I20260619 16:43:44.120759 5032 catalog_manager.cc:5495] T 203c077ee85d481bb4ed7d9d496a2c47 P 14daf912eac14e86bd37f419caa15787 reported cstate change: term changed from 0 to 1, leader changed from <none> to 14daf912eac14e86bd37f419caa15787 (127.0.97.2). New cstate: current_term: 1 leader_uuid: "14daf912eac14e86bd37f419caa15787" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14daf912eac14e86bd37f419caa15787" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44647 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "590f0821b945497ba33007ad423a60b9" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37023 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c2e859f3cb0d46f197282eafbd14fe0f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41581 } health_report { overall_health: UNKNOWN } } }
W20260619 16:43:44.137856 5221 tablet.cc:2369] T 203c077ee85d481bb4ed7d9d496a2c47 P 590f0821b945497ba33007ad423a60b9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 16:43:44.160740 5494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:44.158916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42558 (local address 127.0.97.2:44647)
0619 16:43:44.159109 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0619 16:43:44.159116 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:43:44.159184 (+ 68us) server_negotiation.cc:407] Connection header received
0619 16:43:44.159367 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:43:44.159371 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:43:44.159445 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:43:44.159524 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:43:44.160250 (+ 726us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:43:44.160254 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:43:44.160257 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:43:44.160270 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:43:44.160301 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:43:44.160312 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:43:44.160405 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0619 16:43:44.160459 (+ 54us) server_negotiation.cc:299] Negotiation successful
0619 16:43:44.160500 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260619 16:43:44.168596 5436 raft_consensus.cc:1261] T 203c077ee85d481bb4ed7d9d496a2c47 P c2e859f3cb0d46f197282eafbd14fe0f [term 1 FOLLOWER]: Refusing update from remote peer 14daf912eac14e86bd37f419caa15787: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:43:44.168627 5175 raft_consensus.cc:1261] T 203c077ee85d481bb4ed7d9d496a2c47 P 590f0821b945497ba33007ad423a60b9 [term 1 FOLLOWER]: Refusing update from remote peer 14daf912eac14e86bd37f419caa15787: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:43:44.169601 5510 consensus_queue.cc:1035] T 203c077ee85d481bb4ed7d9d496a2c47 P 14daf912eac14e86bd37f419caa15787 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2e859f3cb0d46f197282eafbd14fe0f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41581 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:43:44.170015 5506 consensus_queue.cc:1035] T 203c077ee85d481bb4ed7d9d496a2c47 P 14daf912eac14e86bd37f419caa15787 [LEADER]: Connected to new peer: Peer: permanent_uuid: "590f0821b945497ba33007ad423a60b9" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37023 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:43:44.185149 5513 mvcc.cc:205] Tried to move back new op lower bound from 7298610889384570880 to 7298610889185689600. Current Snapshot: MvccSnapshot[applied={T|T < 7298610889384570880}]
I20260619 16:43:44.198400 5032 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:43330:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:43:44.198623 5032 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:43330:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:43:44.201577 5032 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3e562a5903ca4065972de95aa01bbb47: Sending DeleteTablet for 3 replicas of tablet 203c077ee85d481bb4ed7d9d496a2c47
I20260619 16:43:44.202800 5155 tablet_service.cc:1514] Processing DeleteTablet for tablet 203c077ee85d481bb4ed7d9d496a2c47 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:43:44 UTC) from {username='slave'} at 127.0.0.1:52820
I20260619 16:43:44.202800 5286 tablet_service.cc:1514] Processing DeleteTablet for tablet 203c077ee85d481bb4ed7d9d496a2c47 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:43:44 UTC) from {username='slave'} at 127.0.0.1:42554
I20260619 16:43:44.203538 5416 tablet_service.cc:1514] Processing DeleteTablet for tablet 203c077ee85d481bb4ed7d9d496a2c47 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:43:44 UTC) from {username='slave'} at 127.0.0.1:53944
I20260619 16:43:44.203866 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu pid 5093
I20260619 16:43:44.204445 5525 tablet_replica.cc:330] T 203c077ee85d481bb4ed7d9d496a2c47 P c2e859f3cb0d46f197282eafbd14fe0f: stopping tablet replica
I20260619 16:43:44.204973 5525 raft_consensus.cc:2229] T 203c077ee85d481bb4ed7d9d496a2c47 P c2e859f3cb0d46f197282eafbd14fe0f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 16:43:44.206002 5525 raft_consensus.cc:2258] T 203c077ee85d481bb4ed7d9d496a2c47 P c2e859f3cb0d46f197282eafbd14fe0f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:43:44.208875 5496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:44.205096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52834 (local address 127.0.97.1:37023)
0619 16:43:44.205292 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0619 16:43:44.205298 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:43:44.206842 (+ 1544us) server_negotiation.cc:407] Connection header received
0619 16:43:44.207043 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:43:44.207048 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:43:44.207171 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:43:44.207271 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:43:44.208200 (+ 929us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:43:44.208206 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:43:44.208212 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:43:44.208232 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:43:44.208268 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:43:44.208284 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:43:44.208400 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0619 16:43:44.208497 (+ 97us) server_negotiation.cc:299] Negotiation successful
0619 16:43:44.208557 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260619 16:43:44.209556 5215 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:43:44.209681 5525 ts_tablet_manager.cc:1905] T 203c077ee85d481bb4ed7d9d496a2c47 P c2e859f3cb0d46f197282eafbd14fe0f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:43:44.216536 5525 ts_tablet_manager.cc:1918] T 203c077ee85d481bb4ed7d9d496a2c47 P c2e859f3cb0d46f197282eafbd14fe0f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 16:43:44.216682 5525 log.cc:1196] T 203c077ee85d481bb4ed7d9d496a2c47 P c2e859f3cb0d46f197282eafbd14fe0f: Deleting WAL directory at /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781887385903558-388-0/minicluster-data/ts-2/wal/wals/203c077ee85d481bb4ed7d9d496a2c47
I20260619 16:43:44.217146 5525 ts_tablet_manager.cc:1939] T 203c077ee85d481bb4ed7d9d496a2c47 P c2e859f3cb0d46f197282eafbd14fe0f: Deleting consensus metadata
I20260619 16:43:44.218896 5019 catalog_manager.cc:4841] TS c2e859f3cb0d46f197282eafbd14fe0f (127.0.97.3:41581): tablet 203c077ee85d481bb4ed7d9d496a2c47 (table test-table [id=e04bd8d791104896b636564129874fc1]) successfully deleted
I20260619 16:43:44.222205 5523 tablet_replica.cc:330] T 203c077ee85d481bb4ed7d9d496a2c47 P 14daf912eac14e86bd37f419caa15787: stopping tablet replica
I20260619 16:43:44.222623 5523 raft_consensus.cc:2229] T 203c077ee85d481bb4ed7d9d496a2c47 P 14daf912eac14e86bd37f419caa15787 [term 1 LEADER]: Raft consensus shutting down.
I20260619 16:43:44.223201 5523 raft_consensus.cc:2258] T 203c077ee85d481bb4ed7d9d496a2c47 P 14daf912eac14e86bd37f419caa15787 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:43:44.224941 5523 ts_tablet_manager.cc:1905] T 203c077ee85d481bb4ed7d9d496a2c47 P 14daf912eac14e86bd37f419caa15787: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:43:44.229889 5523 ts_tablet_manager.cc:1918] T 203c077ee85d481bb4ed7d9d496a2c47 P 14daf912eac14e86bd37f419caa15787: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 16:43:44.230043 5523 log.cc:1196] T 203c077ee85d481bb4ed7d9d496a2c47 P 14daf912eac14e86bd37f419caa15787: Deleting WAL directory at /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781887385903558-388-0/minicluster-data/ts-1/wal/wals/203c077ee85d481bb4ed7d9d496a2c47
I20260619 16:43:44.230446 5523 ts_tablet_manager.cc:1939] T 203c077ee85d481bb4ed7d9d496a2c47 P 14daf912eac14e86bd37f419caa15787: Deleting consensus metadata
I20260619 16:43:44.231770 5019 catalog_manager.cc:4841] TS 14daf912eac14e86bd37f419caa15787 (127.0.97.2:44647): tablet 203c077ee85d481bb4ed7d9d496a2c47 (table test-table [id=e04bd8d791104896b636564129874fc1]) successfully deleted
I20260619 16:43:44.327632 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu with pid 5093
W20260619 16:43:44.346853 5018 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.97.1:37023 (error 108)
I20260619 16:43:44.347532 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu pid 5224
I20260619 16:43:44.348745 5493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:44.348234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:37023 (local address 127.0.0.1:52846)
0619 16:43:44.348474 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0619 16:43:44.348582 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:37023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":148}
W20260619 16:43:44.348975 5018 catalog_manager.cc:4584] TS 590f0821b945497ba33007ad423a60b9 (127.0.97.1:37023): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 203c077ee85d481bb4ed7d9d496a2c47: Network error: Client connection negotiation failed: client connection to 127.0.97.1:37023: connect: Connection refused (error 111)
I20260619 16:43:44.350752 5494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:44.348256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42566 (local address 127.0.97.2:44647)
0619 16:43:44.348437 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0619 16:43:44.348443 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:43:44.348552 (+ 109us) server_negotiation.cc:407] Connection header received
0619 16:43:44.348769 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:43:44.348775 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:43:44.348846 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:43:44.348952 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:43:44.349968 (+ 1016us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:43:44.349974 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:43:44.349979 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:43:44.349998 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:43:44.350036 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:43:44.350052 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:43:44.350187 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0619 16:43:44.350239 (+ 52us) server_negotiation.cc:299] Negotiation successful
0619 16:43:44.350296 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260619 16:43:44.351485 5346 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:43:44.402463 5493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:44.401949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:37023 (local address 127.0.0.1:52856)
0619 16:43:44.402186 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0619 16:43:44.402300 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:37023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260619 16:43:44.403508 5493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:44.403129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:37023 (local address 127.0.0.1:52862)
0619 16:43:44.403268 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0619 16:43:44.403349 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:37023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260619 16:43:44.455614 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu with pid 5224
I20260619 16:43:44.464056 5493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:44.463512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:37023 (local address 127.0.0.1:52874)
0619 16:43:44.463749 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0619 16:43:44.463871 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:37023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260619 16:43:44.465059 5493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:44.464671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:37023 (local address 127.0.0.1:52880)
0619 16:43:44.464816 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0619 16:43:44.464896 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:37023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260619 16:43:44.469373 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu pid 5354
I20260619 16:43:44.471817 5497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:44.469996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53954 (local address 127.0.97.3:41581)
0619 16:43:44.470180 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0619 16:43:44.470186 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:43:44.470204 (+ 18us) server_negotiation.cc:407] Connection header received
0619 16:43:44.470332 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:43:44.470337 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:43:44.470405 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:43:44.470485 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:43:44.471262 (+ 777us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:43:44.471268 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:43:44.471272 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:43:44.471286 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:43:44.471314 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:43:44.471325 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:43:44.471430 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0619 16:43:44.471511 (+ 81us) server_negotiation.cc:299] Negotiation successful
0619 16:43:44.471565 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260619 16:43:44.472342 5476 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:43:44.545579 5493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:44.545070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:37023 (local address 127.0.0.1:52888)
0619 16:43:44.545300 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0619 16:43:44.545406 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:37023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260619 16:43:44.546705 5493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:44.546282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:37023 (local address 127.0.0.1:52896)
0619 16:43:44.546475 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0619 16:43:44.546559 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:37023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
I20260619 16:43:44.556445 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu with pid 5354
I20260619 16:43:44.568781 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu pid 5003
I20260619 16:43:44.569846 5064 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:43:44.671483 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu with pid 5003
2026-06-19T16:43:44Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (2475 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260619 16:43:44.706506 388 mini_oidc.cc:152] Starting JWKS server
I20260619 16:43:44.708824 388 webserver.cc:460] Webserver started at https://127.0.0.1:43471/ using document root <none> and password file <none>
I20260619 16:43:44.709074 388 mini_oidc.cc:189] Starting OIDC Discovery server
I20260619 16:43:44.709643 388 webserver.cc:460] Webserver started at http://127.0.0.1:38153/ using document root <none> and password file <none>
2026-06-19T16:43:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T16:43:44Z Disabled control of system clock
I20260619 16:43:44.732420 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu
/tmp/dist-test-taskEs2EH8/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781887385903558-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781887385903558-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781887385903558-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781887385903558-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:40339
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:35899
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:40339
--enable_jwt_token_auth=true
--jwks_url=https://localhost:43471/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781887385903558-388-0/testchainca.cert with env {}
W20260619 16:43:45.031033 5537 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:43:45.031311 5537 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:43:45.031396 5537 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:43:45.038424 5537 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:43:45.038525 5537 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:43:45.038569 5537 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:43:45.038622 5537 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:43:45.038664 5537 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260619 16:43:45.038722 5537 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260619 16:43:45.038758 5537 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:43471/jwks.json
I20260619 16:43:45.048321 5537 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35899
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781887385903558-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781887385903558-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:40339
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:40339
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:43471/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781887385903558-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781887385903558-388-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781887385903558-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:43:45.049513 5537 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:43:45.050715 5537 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:43:45.061395 5545 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:43:45.061473 5542 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:43:45.061429 5543 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:43:45.062317 5537 server_base.cc:1034] running on GCE node
I20260619 16:43:45.063005 5537 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:43:45.064340 5537 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:43:45.065541 5537 hybrid_clock.cc:648] HybridClock initialized: now 1781887425065500 us; error 35 us; skew 500 ppm
I20260619 16:43:45.065900 5537 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 16:43:45.067934 5537 webserver.cc:460] Webserver started at http://127.0.97.62:39957/ using document root <none> and password file <none>
I20260619 16:43:45.068573 5537 fs_manager.cc:359] Metadata directory not provided
I20260619 16:43:45.068665 5537 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:43:45.068948 5537 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:43:45.071681 5537 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781887385903558-388-0/minicluster-data/master-0/data/instance:
uuid: "eadc4bddfca54125a468739af03713b4"
format_stamp: "Formatted at 2026-06-19 16:43:45 on dist-test-slave-wvz7"
server_key: "f3d8cb60fb1e09b5c5a7f9730534b6db"
server_key_iv: "68536b06270721eacf17a1824d5a68ab"
server_key_version: "encryptionkey@0"
I20260619 16:43:45.072427 5537 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781887385903558-388-0/minicluster-data/master-0/wal/instance:
uuid: "eadc4bddfca54125a468739af03713b4"
format_stamp: "Formatted at 2026-06-19 16:43:45 on dist-test-slave-wvz7"
server_key: "f3d8cb60fb1e09b5c5a7f9730534b6db"
server_key_iv: "68536b06270721eacf17a1824d5a68ab"
server_key_version: "encryptionkey@0"
I20260619 16:43:45.077278 5537 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260619 16:43:45.080626 5551 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:43:45.081990 5537 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 16:43:45.082149 5537 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781887385903558-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781887385903558-388-0/minicluster-data/master-0/wal
uuid: "eadc4bddfca54125a468739af03713b4"
format_stamp: "Formatted at 2026-06-19 16:43:45 on dist-test-slave-wvz7"
server_key: "f3d8cb60fb1e09b5c5a7f9730534b6db"
server_key_iv: "68536b06270721eacf17a1824d5a68ab"
server_key_version: "encryptionkey@0"
I20260619 16:43:45.082329 5537 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781887385903558-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781887385903558-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781887385903558-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:43:45.109202 5537 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:43:45.110080 5537 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:43:45.110355 5537 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:43:45.126873 5537 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:40339
I20260619 16:43:45.126895 5602 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:40339 every 8 connection(s)
I20260619 16:43:45.128119 5537 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781887385903558-388-0/minicluster-data/master-0/data/info.pb
I20260619 16:43:45.132027 5603 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:43:45.137444 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu as pid 5537
I20260619 16:43:45.137615 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781887385903558-388-0/minicluster-data/master-0/wal/instance
I20260619 16:43:45.138193 388 external_mini_cluster.cc:1428] Setting key d9f2e14ad134239fef8dd3592f1e9cf1
I20260619 16:43:45.142735 5603 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eadc4bddfca54125a468739af03713b4: Bootstrap starting.
I20260619 16:43:45.146027 5603 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P eadc4bddfca54125a468739af03713b4: Neither blocks nor log segments found. Creating new log.
I20260619 16:43:45.147166 5603 log.cc:824] T 00000000000000000000000000000000 P eadc4bddfca54125a468739af03713b4: Log is configured to *not* fsync() on all Append() calls
I20260619 16:43:45.149976 5603 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eadc4bddfca54125a468739af03713b4: No bootstrap required, opened a new log
I20260619 16:43:45.150928 5606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:45.139964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56984 (local address 127.0.97.62:40339)
0619 16:43:45.140917 (+ 953us) server_negotiation.cc:206] Beginning negotiation
0619 16:43:45.140928 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 16:43:45.140969 (+ 41us) server_negotiation.cc:407] Connection header received
0619 16:43:45.142563 (+ 1594us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:43:45.142586 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:43:45.143044 (+ 458us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:43:45.143353 (+ 309us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:43:45.143899 (+ 546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:45.145148 (+ 1249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:45.146898 (+ 1750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:45.147585 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:45.147819 (+ 234us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:43:45.148808 (+ 989us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:43:45.148816 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:43:45.148829 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:43:45.149320 (+ 491us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:43:45.149390 (+ 70us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:43:45.149413 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:43:45.149490 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0619 16:43:45.149728 (+ 238us) server_negotiation.cc:299] Negotiation successful
0619 16:43:45.150061 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":533,"thread_start_us":322,"threads_started":1}
I20260619 16:43:45.154688 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 16:43:45.158143 5603 raft_consensus.cc:348] T 00000000000000000000000000000000 P eadc4bddfca54125a468739af03713b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eadc4bddfca54125a468739af03713b4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40339 } }
I20260619 16:43:45.158452 5603 raft_consensus.cc:374] T 00000000000000000000000000000000 P eadc4bddfca54125a468739af03713b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:43:45.158547 5603 raft_consensus.cc:729] T 00000000000000000000000000000000 P eadc4bddfca54125a468739af03713b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eadc4bddfca54125a468739af03713b4, State: Initialized, Role: FOLLOWER
I20260619 16:43:45.159096 5603 consensus_queue.cc:260] T 00000000000000000000000000000000 P eadc4bddfca54125a468739af03713b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eadc4bddfca54125a468739af03713b4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40339 } }
I20260619 16:43:45.159294 5603 raft_consensus.cc:388] T 00000000000000000000000000000000 P eadc4bddfca54125a468739af03713b4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:43:45.159405 5603 raft_consensus.cc:482] T 00000000000000000000000000000000 P eadc4bddfca54125a468739af03713b4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:43:45.159538 5603 raft_consensus.cc:3037] T 00000000000000000000000000000000 P eadc4bddfca54125a468739af03713b4 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:43:45.161852 5603 raft_consensus.cc:504] T 00000000000000000000000000000000 P eadc4bddfca54125a468739af03713b4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eadc4bddfca54125a468739af03713b4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40339 } }
I20260619 16:43:45.162322 5603 leader_election.cc:302] T 00000000000000000000000000000000 P eadc4bddfca54125a468739af03713b4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: eadc4bddfca54125a468739af03713b4; no voters:
I20260619 16:43:45.162854 5603 leader_election.cc:288] T 00000000000000000000000000000000 P eadc4bddfca54125a468739af03713b4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 16:43:45.162983 5608 raft_consensus.cc:2781] T 00000000000000000000000000000000 P eadc4bddfca54125a468739af03713b4 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:43:45.163576 5608 raft_consensus.cc:686] T 00000000000000000000000000000000 P eadc4bddfca54125a468739af03713b4 [term 1 LEADER]: Becoming Leader. State: Replica: eadc4bddfca54125a468739af03713b4, State: Running, Role: LEADER
I20260619 16:43:45.164081 5608 consensus_queue.cc:237] T 00000000000000000000000000000000 P eadc4bddfca54125a468739af03713b4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eadc4bddfca54125a468739af03713b4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40339 } }
I20260619 16:43:45.164713 5603 sys_catalog.cc:564] T 00000000000000000000000000000000 P eadc4bddfca54125a468739af03713b4 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:43:45.166760 5609 sys_catalog.cc:455] T 00000000000000000000000000000000 P eadc4bddfca54125a468739af03713b4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "eadc4bddfca54125a468739af03713b4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eadc4bddfca54125a468739af03713b4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40339 } } }
I20260619 16:43:45.166772 5610 sys_catalog.cc:455] T 00000000000000000000000000000000 P eadc4bddfca54125a468739af03713b4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader eadc4bddfca54125a468739af03713b4. Latest consensus state: current_term: 1 leader_uuid: "eadc4bddfca54125a468739af03713b4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eadc4bddfca54125a468739af03713b4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40339 } } }
I20260619 16:43:45.167162 5610 sys_catalog.cc:458] T 00000000000000000000000000000000 P eadc4bddfca54125a468739af03713b4 [sys.catalog]: This master's current role is: LEADER
I20260619 16:43:45.167160 5609 sys_catalog.cc:458] T 00000000000000000000000000000000 P eadc4bddfca54125a468739af03713b4 [sys.catalog]: This master's current role is: LEADER
I20260619 16:43:45.171350 5615 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:43:45.176707 5615 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:43:45.184595 5615 catalog_manager.cc:1347] Generated new cluster ID: 7f4121b8000c40249a2ae35bf4ddfed4
I20260619 16:43:45.184710 5615 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:43:45.203694 5615 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 16:43:45.205400 5615 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:43:45.214421 5615 catalog_manager.cc:5868] T 00000000000000000000000000000000 P eadc4bddfca54125a468739af03713b4: Generated new TSK 0
I20260619 16:43:45.215381 5615 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260619 16:43:45.261668 5633 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260619 16:43:45.261854 5633 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260619 16:43:45.262574 5606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:45.234738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56994 (local address 127.0.97.62:40339)
0619 16:43:45.234952 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0619 16:43:45.234958 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:43:45.235348 (+ 390us) server_negotiation.cc:407] Connection header received
0619 16:43:45.235665 (+ 317us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:43:45.235669 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:43:45.235748 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:43:45.235854 (+ 106us) server_negotiation.cc:226] Negotiated authn=JWT
0619 16:43:45.236424 (+ 570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:45.237146 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:45.239153 (+ 2007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:45.239612 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:45.239705 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:43:45.240224 (+ 519us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0619 16:43:45.262093 (+ 21869us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43471/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0619 16:43:45.262365 (+ 272us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:56994: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43471/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":67}
W20260619 16:43:45.262691 5606 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:56994: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43471/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260619 16:43:45.264356 5632 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:43:45.234628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40339 (local address 127.0.0.1:56994)
0619 16:43:45.235246 (+ 618us) negotiation.cc:107] Waiting for socket to connect
0619 16:43:45.235265 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 16:43:45.235435 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:43:45.235913 (+ 478us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:43:45.235918 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:43:45.235932 (+ 14us) client_negotiation.cc:189] Negotiated authn=JWT
0619 16:43:45.236272 (+ 340us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:43:45.236283 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:43:45.237273 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:43:45.237279 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:43:45.239017 (+ 1738us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:43:45.239033 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:43:45.239718 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:43:45.239759 (+ 41us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:43:45.239943 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:43:45.240011 (+ 68us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0619 16:43:45.262359 (+ 22348us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43471/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0619 16:43:45.262696 (+ 337us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.97.62:40339: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43471/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":434,"thread_start_us":304,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.0.97.62:40339: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43471/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260619 16:43:45.266696 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu pid 5537
I20260619 16:43:45.267735 5598 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:43:45.357937 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu with pid 5537
2026-06-19T16:43:45Z chronyd exiting
I20260619 16:43:45.394343 388 test_util.cc:175] -----------------------------------------------
I20260619 16:43:45.394471 388 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781887385903558-388-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (692 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260619 16:43:45.398029 388 mini_oidc.cc:152] Starting JWKS server
I20260619 16:43:45.399713 388 webserver.cc:460] Webserver started at https://127.0.0.1:45385/ using document root <none> and password file <none>
I20260619 16:43:45.399891 388 mini_oidc.cc:189] Starting OIDC Discovery server
I20260619 16:43:45.400446 388 webserver.cc:460] Webserver started at http://127.0.0.1:41457/ using document root <none> and password file <none>
2026-06-19T16:43:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T16:43:45Z Disabled control of system clock
I20260619 16:43:45.423151 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu
/tmp/dist-test-taskEs2EH8/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781887385903558-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781887385903558-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781887385903558-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781887385903558-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34051
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:45671
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:34051
--enable_jwt_token_auth=true
--jwks_url=https://localhost:45385/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781887385903558-388-0/testchainca.cert with env {}
W20260619 16:43:45.721055 5640 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:43:45.721333 5640 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:43:45.721418 5640 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:43:45.728471 5640 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:43:45.728575 5640 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:43:45.728610 5640 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:43:45.728660 5640 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:43:45.728703 5640 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260619 16:43:45.728760 5640 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260619 16:43:45.728799 5640 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:45385/jwks.json
I20260619 16:43:45.738498 5640 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45671
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781887385903558-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781887385903558-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:34051
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34051
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:45385/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781887385903558-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781887385903558-388-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781887385903558-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:43:45.739660 5640 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:43:45.740839 5640 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:43:45.751557 5646 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:43:45.751586 5645 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:43:45.751719 5648 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:43:45.752156 5640 server_base.cc:1034] running on GCE node
I20260619 16:43:45.752862 5640 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:43:45.754266 5640 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:43:45.755482 5640 hybrid_clock.cc:648] HybridClock initialized: now 1781887425755442 us; error 35 us; skew 500 ppm
I20260619 16:43:45.755815 5640 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 16:43:45.757947 5640 webserver.cc:460] Webserver started at http://127.0.97.62:43535/ using document root <none> and password file <none>
I20260619 16:43:45.758592 5640 fs_manager.cc:359] Metadata directory not provided
I20260619 16:43:45.758680 5640 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:43:45.758978 5640 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:43:45.761698 5640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781887385903558-388-0/minicluster-data/master-0/data/instance:
uuid: "318938b3145149e08455d4ca610c5dbb"
format_stamp: "Formatted at 2026-06-19 16:43:45 on dist-test-slave-wvz7"
server_key: "73b4bfafab0df5e70286f8ee92292915"
server_key_iv: "e5c89000c45e7f86bb4007b329fc289d"
server_key_version: "encryptionkey@0"
I20260619 16:43:45.762487 5640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781887385903558-388-0/minicluster-data/master-0/wal/instance:
uuid: "318938b3145149e08455d4ca610c5dbb"
format_stamp: "Formatted at 2026-06-19 16:43:45 on dist-test-slave-wvz7"
server_key: "73b4bfafab0df5e70286f8ee92292915"
server_key_iv: "e5c89000c45e7f86bb4007b329fc289d"
server_key_version: "encryptionkey@0"
I20260619 16:43:45.767272 5640 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260619 16:43:45.770831 5654 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:43:45.772200 5640 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 16:43:45.772359 5640 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781887385903558-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781887385903558-388-0/minicluster-data/master-0/wal
uuid: "318938b3145149e08455d4ca610c5dbb"
format_stamp: "Formatted at 2026-06-19 16:43:45 on dist-test-slave-wvz7"
server_key: "73b4bfafab0df5e70286f8ee92292915"
server_key_iv: "e5c89000c45e7f86bb4007b329fc289d"
server_key_version: "encryptionkey@0"
I20260619 16:43:45.772544 5640 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781887385903558-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781887385903558-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781887385903558-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:43:45.787986 5640 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:43:45.788806 5640 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:43:45.789062 5640 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:43:45.805773 5640 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:34051
I20260619 16:43:45.805810 5705 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:34051 every 8 connection(s)
I20260619 16:43:45.807010 5640 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781887385903558-388-0/minicluster-data/master-0/data/info.pb
I20260619 16:43:45.809016 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu as pid 5640
I20260619 16:43:45.809185 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781887385903558-388-0/minicluster-data/master-0/wal/instance
I20260619 16:43:45.809788 388 external_mini_cluster.cc:1428] Setting key 599e95858127dfcd28acd2c4b803033f
I20260619 16:43:45.810716 5706 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:43:45.822613 5709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:45.811485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33438 (local address 127.0.97.62:34051)
0619 16:43:45.812364 (+ 879us) server_negotiation.cc:206] Beginning negotiation
0619 16:43:45.812375 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 16:43:45.812415 (+ 40us) server_negotiation.cc:407] Connection header received
0619 16:43:45.813864 (+ 1449us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:43:45.813885 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:43:45.814349 (+ 464us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:43:45.814665 (+ 316us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:43:45.815264 (+ 599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:45.816619 (+ 1355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:45.818446 (+ 1827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:45.819102 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:45.819228 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:43:45.820396 (+ 1168us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:43:45.820403 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:43:45.820416 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:43:45.820976 (+ 560us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:43:45.821044 (+ 68us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:43:45.821064 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:43:45.821150 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0619 16:43:45.821359 (+ 209us) server_negotiation.cc:299] Negotiation successful
0619 16:43:45.821666 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":503,"thread_start_us":306,"threads_started":1}
I20260619 16:43:45.823669 5706 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 318938b3145149e08455d4ca610c5dbb: Bootstrap starting.
I20260619 16:43:45.826207 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 16:43:45.826835 5706 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 318938b3145149e08455d4ca610c5dbb: Neither blocks nor log segments found. Creating new log.
I20260619 16:43:45.827739 5706 log.cc:824] T 00000000000000000000000000000000 P 318938b3145149e08455d4ca610c5dbb: Log is configured to *not* fsync() on all Append() calls
I20260619 16:43:45.830492 5706 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 318938b3145149e08455d4ca610c5dbb: No bootstrap required, opened a new log
I20260619 16:43:45.837723 5706 raft_consensus.cc:348] T 00000000000000000000000000000000 P 318938b3145149e08455d4ca610c5dbb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "318938b3145149e08455d4ca610c5dbb" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34051 } }
I20260619 16:43:45.838137 5706 raft_consensus.cc:374] T 00000000000000000000000000000000 P 318938b3145149e08455d4ca610c5dbb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:43:45.838235 5706 raft_consensus.cc:729] T 00000000000000000000000000000000 P 318938b3145149e08455d4ca610c5dbb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 318938b3145149e08455d4ca610c5dbb, State: Initialized, Role: FOLLOWER
I20260619 16:43:45.838779 5706 consensus_queue.cc:260] T 00000000000000000000000000000000 P 318938b3145149e08455d4ca610c5dbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "318938b3145149e08455d4ca610c5dbb" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34051 } }
I20260619 16:43:45.838989 5706 raft_consensus.cc:388] T 00000000000000000000000000000000 P 318938b3145149e08455d4ca610c5dbb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:43:45.839107 5706 raft_consensus.cc:482] T 00000000000000000000000000000000 P 318938b3145149e08455d4ca610c5dbb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:43:45.839259 5706 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 318938b3145149e08455d4ca610c5dbb [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:43:45.841289 5706 raft_consensus.cc:504] T 00000000000000000000000000000000 P 318938b3145149e08455d4ca610c5dbb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "318938b3145149e08455d4ca610c5dbb" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34051 } }
I20260619 16:43:45.841842 5706 leader_election.cc:302] T 00000000000000000000000000000000 P 318938b3145149e08455d4ca610c5dbb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 318938b3145149e08455d4ca610c5dbb; no voters:
I20260619 16:43:45.842417 5706 leader_election.cc:288] T 00000000000000000000000000000000 P 318938b3145149e08455d4ca610c5dbb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 16:43:45.842531 5711 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 318938b3145149e08455d4ca610c5dbb [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:43:45.844229 5706 sys_catalog.cc:564] T 00000000000000000000000000000000 P 318938b3145149e08455d4ca610c5dbb [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:43:45.844584 5711 raft_consensus.cc:686] T 00000000000000000000000000000000 P 318938b3145149e08455d4ca610c5dbb [term 1 LEADER]: Becoming Leader. State: Replica: 318938b3145149e08455d4ca610c5dbb, State: Running, Role: LEADER
I20260619 16:43:45.845046 5711 consensus_queue.cc:237] T 00000000000000000000000000000000 P 318938b3145149e08455d4ca610c5dbb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "318938b3145149e08455d4ca610c5dbb" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34051 } }
I20260619 16:43:45.847829 5712 sys_catalog.cc:455] T 00000000000000000000000000000000 P 318938b3145149e08455d4ca610c5dbb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "318938b3145149e08455d4ca610c5dbb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "318938b3145149e08455d4ca610c5dbb" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34051 } } }
I20260619 16:43:45.847807 5713 sys_catalog.cc:455] T 00000000000000000000000000000000 P 318938b3145149e08455d4ca610c5dbb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 318938b3145149e08455d4ca610c5dbb. Latest consensus state: current_term: 1 leader_uuid: "318938b3145149e08455d4ca610c5dbb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "318938b3145149e08455d4ca610c5dbb" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34051 } } }
I20260619 16:43:45.848265 5712 sys_catalog.cc:458] T 00000000000000000000000000000000 P 318938b3145149e08455d4ca610c5dbb [sys.catalog]: This master's current role is: LEADER
I20260619 16:43:45.848268 5713 sys_catalog.cc:458] T 00000000000000000000000000000000 P 318938b3145149e08455d4ca610c5dbb [sys.catalog]: This master's current role is: LEADER
I20260619 16:43:45.850327 5723 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:43:45.854954 5723 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:43:45.863631 5723 catalog_manager.cc:1347] Generated new cluster ID: e07e7770e5c54448a780c219ad4fb9e2
I20260619 16:43:45.863814 5723 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:43:45.903167 5723 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 16:43:45.904825 5723 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:43:45.916810 5723 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 318938b3145149e08455d4ca610c5dbb: Generated new TSK 0
I20260619 16:43:45.917652 5723 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260619 16:43:45.952397 5736 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260619 16:43:45.952525 5736 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260619 16:43:45.953596 5709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:45.933689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33440 (local address 127.0.97.62:34051)
0619 16:43:45.933947 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0619 16:43:45.933954 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:43:45.934379 (+ 425us) server_negotiation.cc:407] Connection header received
0619 16:43:45.934610 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:43:45.934614 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:43:45.934693 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:43:45.934830 (+ 137us) server_negotiation.cc:226] Negotiated authn=JWT
0619 16:43:45.935361 (+ 531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:45.936081 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:45.938095 (+ 2014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:45.938549 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:45.938653 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:43:45.939217 (+ 564us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0619 16:43:45.952952 (+ 13735us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45385/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0619 16:43:45.953296 (+ 344us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:33440: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45385/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":104}
W20260619 16:43:45.953717 5709 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:33440: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45385/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260619 16:43:45.954030 5735 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:43:45.933581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34051 (local address 127.0.0.1:33440)
0619 16:43:45.934271 (+ 690us) negotiation.cc:107] Waiting for socket to connect
0619 16:43:45.934292 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 16:43:45.934451 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:43:45.934871 (+ 420us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:43:45.934876 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:43:45.934889 (+ 13us) client_negotiation.cc:189] Negotiated authn=JWT
0619 16:43:45.935220 (+ 331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:43:45.935232 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:43:45.936217 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:43:45.936223 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:43:45.937961 (+ 1738us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:43:45.937976 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:43:45.938663 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:43:45.938732 (+ 69us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:43:45.938918 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:43:45.938966 (+ 48us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0619 16:43:45.953216 (+ 14250us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45385/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0619 16:43:45.953525 (+ 309us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.97.62:34051: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45385/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":521,"thread_start_us":331,"threads_started":1}
I20260619 16:43:45.955190 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu pid 5640
I20260619 16:43:45.956455 5701 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:43:46.047715 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu with pid 5640
2026-06-19T16:43:46Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (701 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260619 16:43:46.099864 388 mini_oidc.cc:152] Starting JWKS server
I20260619 16:43:46.101634 388 webserver.cc:460] Webserver started at https://127.0.0.1:41231/ using document root <none> and password file <none>
I20260619 16:43:46.101862 388 mini_oidc.cc:189] Starting OIDC Discovery server
I20260619 16:43:46.102476 388 webserver.cc:460] Webserver started at http://127.0.0.1:41511/ using document root <none> and password file <none>
2026-06-19T16:43:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T16:43:46Z Disabled control of system clock
I20260619 16:43:46.125468 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu
/tmp/dist-test-taskEs2EH8/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781887385903558-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781887385903558-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781887385903558-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781887385903558-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:41715
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:46501
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:41715
--enable_jwt_token_auth=true
--jwks_url=https://localhost:41231/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260619 16:43:46.423877 5743 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:43:46.424180 5743 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:43:46.424311 5743 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:43:46.431572 5743 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:43:46.431713 5743 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:43:46.431783 5743 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:43:46.431874 5743 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:43:46.431949 5743 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260619 16:43:46.432042 5743 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260619 16:43:46.432101 5743 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:41231/jwks.json
I20260619 16:43:46.441653 5743 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46501
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781887385903558-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781887385903558-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:41715
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:41715
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:41231/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781887385903558-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781887385903558-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:43:46.442938 5743 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:43:46.444579 5743 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:43:46.455225 5748 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:43:46.455199 5749 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:43:46.455674 5751 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:43:46.456672 5743 server_base.cc:1034] running on GCE node
I20260619 16:43:46.457381 5743 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:43:46.458725 5743 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:43:46.459924 5743 hybrid_clock.cc:648] HybridClock initialized: now 1781887426459879 us; error 33 us; skew 500 ppm
I20260619 16:43:46.460237 5743 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 16:43:46.465317 5743 webserver.cc:460] Webserver started at http://127.0.97.62:32951/ using document root <none> and password file <none>
I20260619 16:43:46.466029 5743 fs_manager.cc:359] Metadata directory not provided
I20260619 16:43:46.466127 5743 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:43:46.466418 5743 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:43:46.469132 5743 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781887385903558-388-0/minicluster-data/master-0/data/instance:
uuid: "88eb643260a24550b9f4c04047c81805"
format_stamp: "Formatted at 2026-06-19 16:43:46 on dist-test-slave-wvz7"
server_key: "53311e968355c2a4b3bc9958e48a13fe"
server_key_iv: "a3d7453ffee0835fb692b42ec0000f96"
server_key_version: "encryptionkey@0"
I20260619 16:43:46.469938 5743 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781887385903558-388-0/minicluster-data/master-0/wal/instance:
uuid: "88eb643260a24550b9f4c04047c81805"
format_stamp: "Formatted at 2026-06-19 16:43:46 on dist-test-slave-wvz7"
server_key: "53311e968355c2a4b3bc9958e48a13fe"
server_key_iv: "a3d7453ffee0835fb692b42ec0000f96"
server_key_version: "encryptionkey@0"
I20260619 16:43:46.474798 5743 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260619 16:43:46.478145 5757 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:43:46.479498 5743 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260619 16:43:46.479673 5743 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781887385903558-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781887385903558-388-0/minicluster-data/master-0/wal
uuid: "88eb643260a24550b9f4c04047c81805"
format_stamp: "Formatted at 2026-06-19 16:43:46 on dist-test-slave-wvz7"
server_key: "53311e968355c2a4b3bc9958e48a13fe"
server_key_iv: "a3d7453ffee0835fb692b42ec0000f96"
server_key_version: "encryptionkey@0"
I20260619 16:43:46.479872 5743 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781887385903558-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781887385903558-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781887385903558-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:43:46.492826 5743 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:43:46.493633 5743 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:43:46.493902 5743 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:43:46.510167 5743 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:41715
I20260619 16:43:46.510179 5808 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:41715 every 8 connection(s)
I20260619 16:43:46.511471 5743 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781887385903558-388-0/minicluster-data/master-0/data/info.pb
I20260619 16:43:46.515090 5809 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:43:46.520094 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu as pid 5743
I20260619 16:43:46.520291 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781887385903558-388-0/minicluster-data/master-0/wal/instance
I20260619 16:43:46.520977 388 external_mini_cluster.cc:1428] Setting key 791b34bca97fe88e9996b372cea039d4
I20260619 16:43:46.525761 5809 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 88eb643260a24550b9f4c04047c81805: Bootstrap starting.
I20260619 16:43:46.528928 5809 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 88eb643260a24550b9f4c04047c81805: Neither blocks nor log segments found. Creating new log.
I20260619 16:43:46.530074 5809 log.cc:824] T 00000000000000000000000000000000 P 88eb643260a24550b9f4c04047c81805: Log is configured to *not* fsync() on all Append() calls
I20260619 16:43:46.533048 5809 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 88eb643260a24550b9f4c04047c81805: No bootstrap required, opened a new log
I20260619 16:43:46.533906 5812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:46.522841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37716 (local address 127.0.97.62:41715)
0619 16:43:46.523834 (+ 993us) server_negotiation.cc:206] Beginning negotiation
0619 16:43:46.523851 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0619 16:43:46.523895 (+ 44us) server_negotiation.cc:407] Connection header received
0619 16:43:46.525476 (+ 1581us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:43:46.525496 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:43:46.526124 (+ 628us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:43:46.526448 (+ 324us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:43:46.526920 (+ 472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:46.528183 (+ 1263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:46.529876 (+ 1693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:46.530650 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:46.530829 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:43:46.531790 (+ 961us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:43:46.531797 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:43:46.531810 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:43:46.532301 (+ 491us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:43:46.532370 (+ 69us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:43:46.532389 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:43:46.532498 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0619 16:43:46.532667 (+ 169us) server_negotiation.cc:299] Negotiation successful
0619 16:43:46.532944 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":552,"thread_start_us":345,"threads_started":1}
I20260619 16:43:46.537371 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 16:43:46.541217 5809 raft_consensus.cc:348] T 00000000000000000000000000000000 P 88eb643260a24550b9f4c04047c81805 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88eb643260a24550b9f4c04047c81805" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41715 } }
I20260619 16:43:46.541515 5809 raft_consensus.cc:374] T 00000000000000000000000000000000 P 88eb643260a24550b9f4c04047c81805 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:43:46.541610 5809 raft_consensus.cc:729] T 00000000000000000000000000000000 P 88eb643260a24550b9f4c04047c81805 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88eb643260a24550b9f4c04047c81805, State: Initialized, Role: FOLLOWER
I20260619 16:43:46.542150 5809 consensus_queue.cc:260] T 00000000000000000000000000000000 P 88eb643260a24550b9f4c04047c81805 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88eb643260a24550b9f4c04047c81805" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41715 } }
I20260619 16:43:46.542344 5809 raft_consensus.cc:388] T 00000000000000000000000000000000 P 88eb643260a24550b9f4c04047c81805 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:43:46.542455 5809 raft_consensus.cc:482] T 00000000000000000000000000000000 P 88eb643260a24550b9f4c04047c81805 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:43:46.542627 5809 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 88eb643260a24550b9f4c04047c81805 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:43:46.544593 5809 raft_consensus.cc:504] T 00000000000000000000000000000000 P 88eb643260a24550b9f4c04047c81805 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88eb643260a24550b9f4c04047c81805" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41715 } }
I20260619 16:43:46.545042 5809 leader_election.cc:302] T 00000000000000000000000000000000 P 88eb643260a24550b9f4c04047c81805 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 88eb643260a24550b9f4c04047c81805; no voters:
I20260619 16:43:46.545568 5809 leader_election.cc:288] T 00000000000000000000000000000000 P 88eb643260a24550b9f4c04047c81805 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 16:43:46.545722 5814 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 88eb643260a24550b9f4c04047c81805 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:43:46.546439 5814 raft_consensus.cc:686] T 00000000000000000000000000000000 P 88eb643260a24550b9f4c04047c81805 [term 1 LEADER]: Becoming Leader. State: Replica: 88eb643260a24550b9f4c04047c81805, State: Running, Role: LEADER
I20260619 16:43:46.547010 5814 consensus_queue.cc:237] T 00000000000000000000000000000000 P 88eb643260a24550b9f4c04047c81805 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88eb643260a24550b9f4c04047c81805" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41715 } }
I20260619 16:43:46.547526 5809 sys_catalog.cc:564] T 00000000000000000000000000000000 P 88eb643260a24550b9f4c04047c81805 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:43:46.549831 5815 sys_catalog.cc:455] T 00000000000000000000000000000000 P 88eb643260a24550b9f4c04047c81805 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "88eb643260a24550b9f4c04047c81805" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88eb643260a24550b9f4c04047c81805" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41715 } } }
I20260619 16:43:46.550263 5815 sys_catalog.cc:458] T 00000000000000000000000000000000 P 88eb643260a24550b9f4c04047c81805 [sys.catalog]: This master's current role is: LEADER
I20260619 16:43:46.549856 5816 sys_catalog.cc:455] T 00000000000000000000000000000000 P 88eb643260a24550b9f4c04047c81805 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88eb643260a24550b9f4c04047c81805. Latest consensus state: current_term: 1 leader_uuid: "88eb643260a24550b9f4c04047c81805" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88eb643260a24550b9f4c04047c81805" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41715 } } }
I20260619 16:43:46.550853 5816 sys_catalog.cc:458] T 00000000000000000000000000000000 P 88eb643260a24550b9f4c04047c81805 [sys.catalog]: This master's current role is: LEADER
I20260619 16:43:46.555032 5821 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:43:46.559590 5821 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:43:46.566813 5821 catalog_manager.cc:1347] Generated new cluster ID: c318f390d0e44c7095b091cd3ab6515c
I20260619 16:43:46.566926 5821 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:43:46.602392 5821 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 16:43:46.604259 5821 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:43:46.610387 5821 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 88eb643260a24550b9f4c04047c81805: Generated new TSK 0
I20260619 16:43:46.611382 5821 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 16:43:46.645896 5812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:46.637787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37728 (local address 127.0.97.62:41715)
0619 16:43:46.637978 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0619 16:43:46.637985 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:43:46.638290 (+ 305us) server_negotiation.cc:407] Connection header received
0619 16:43:46.638489 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:43:46.638494 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:43:46.638585 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:43:46.638769 (+ 184us) server_negotiation.cc:226] Negotiated authn=JWT
0619 16:43:46.639158 (+ 389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:46.640252 (+ 1094us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:46.641606 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:46.642308 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:46.642444 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:43:46.642944 (+ 500us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0619 16:43:46.645245 (+ 2301us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41231/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0619 16:43:46.645568 (+ 323us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:37728: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41231/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":69}
W20260619 16:43:46.646044 5812 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:37728: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41231/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260619 16:43:46.647505 5838 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:43:46.637605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41715 (local address 127.0.0.1:37728)
0619 16:43:46.638192 (+ 587us) negotiation.cc:107] Waiting for socket to connect
0619 16:43:46.638211 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 16:43:46.638327 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:43:46.638769 (+ 442us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:43:46.638772 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:43:46.638781 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0619 16:43:46.639024 (+ 243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:43:46.639032 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:43:46.640417 (+ 1385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:43:46.640421 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:43:46.641476 (+ 1055us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:43:46.641488 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:43:46.642432 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:43:46.642490 (+ 58us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:43:46.642646 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:43:46.642680 (+ 34us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0619 16:43:46.645444 (+ 2764us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41231/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0619 16:43:46.647082 (+ 1638us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.97.62:41715: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41231/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":458,"mutex_wait_us":1355,"thread_start_us":329,"threads_started":1}
I20260619 16:43:46.648699 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu pid 5743
I20260619 16:43:46.649853 5804 generic_service.cc:196] Checking for leaks (request via RPC)
E20260619 16:43:46.657415 5839 webserver.cc:533] Webserver: sslize failed:
E20260619 16:43:46.657492 5839 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260619 16:43:46.740127 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu with pid 5743
2026-06-19T16:43:46Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (691 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-06-19T16:43:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T16:43:46Z Disabled control of system clock
I20260619 16:43:46.812671 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu
/tmp/dist-test-taskEs2EH8/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781887385903558-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781887385903558-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781887385903558-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781887385903558-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39395
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:39165
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:39395
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781887385903558-388-0/testchainca.cert with env {}
W20260619 16:43:47.113843 5844 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:43:47.114135 5844 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:43:47.114250 5844 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:43:47.121521 5844 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:43:47.121627 5844 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:43:47.121663 5844 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:43:47.121723 5844 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:43:47.121801 5844 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260619 16:43:47.121865 5844 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260619 16:43:47.121901 5844 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260619 16:43:47.131800 5844 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39165
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781887385903558-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781887385903558-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:39395
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39395
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781887385903558-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781887385903558-388-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781887385903558-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:43:47.133098 5844 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:43:47.134506 5844 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:43:47.145332 5850 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:43:47.145357 5852 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:43:47.145426 5849 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:43:47.151391 5844 server_base.cc:1034] running on GCE node
I20260619 16:43:47.152379 5844 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:43:47.153687 5844 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:43:47.154928 5844 hybrid_clock.cc:648] HybridClock initialized: now 1781887427154885 us; error 36 us; skew 500 ppm
I20260619 16:43:47.155282 5844 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 16:43:47.157388 5844 webserver.cc:460] Webserver started at http://127.0.97.62:32967/ using document root <none> and password file <none>
I20260619 16:43:47.158095 5844 fs_manager.cc:359] Metadata directory not provided
I20260619 16:43:47.158197 5844 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:43:47.158478 5844 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:43:47.161203 5844 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781887385903558-388-0/minicluster-data/master-0/data/instance:
uuid: "769504c4735a4b17843633b4a2f8a630"
format_stamp: "Formatted at 2026-06-19 16:43:47 on dist-test-slave-wvz7"
server_key: "2835d52efebd34c591e3b9da7ecb3c83"
server_key_iv: "42a0b8be437008ba050d4f63acc93bfa"
server_key_version: "encryptionkey@0"
I20260619 16:43:47.161998 5844 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781887385903558-388-0/minicluster-data/master-0/wal/instance:
uuid: "769504c4735a4b17843633b4a2f8a630"
format_stamp: "Formatted at 2026-06-19 16:43:47 on dist-test-slave-wvz7"
server_key: "2835d52efebd34c591e3b9da7ecb3c83"
server_key_iv: "42a0b8be437008ba050d4f63acc93bfa"
server_key_version: "encryptionkey@0"
I20260619 16:43:47.166983 5844 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260619 16:43:47.170329 5858 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:43:47.171685 5844 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 16:43:47.171840 5844 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781887385903558-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781887385903558-388-0/minicluster-data/master-0/wal
uuid: "769504c4735a4b17843633b4a2f8a630"
format_stamp: "Formatted at 2026-06-19 16:43:47 on dist-test-slave-wvz7"
server_key: "2835d52efebd34c591e3b9da7ecb3c83"
server_key_iv: "42a0b8be437008ba050d4f63acc93bfa"
server_key_version: "encryptionkey@0"
I20260619 16:43:47.172045 5844 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781887385903558-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781887385903558-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781887385903558-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:43:47.203346 5844 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:43:47.204173 5844 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:43:47.204437 5844 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:43:47.222182 5844 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:39395
I20260619 16:43:47.222190 5909 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:39395 every 8 connection(s)
I20260619 16:43:47.223531 5844 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781887385903558-388-0/minicluster-data/master-0/data/info.pb
I20260619 16:43:47.227094 5910 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:43:47.228322 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu as pid 5844
I20260619 16:43:47.228506 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781887385903558-388-0/minicluster-data/master-0/wal/instance
I20260619 16:43:47.229272 388 external_mini_cluster.cc:1428] Setting key 021fff04d4971eefbbc993f054e116a9
I20260619 16:43:47.239310 5910 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 769504c4735a4b17843633b4a2f8a630: Bootstrap starting.
I20260619 16:43:47.241631 5913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:47.231172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43760 (local address 127.0.97.62:39395)
0619 16:43:47.232154 (+ 982us) server_negotiation.cc:206] Beginning negotiation
0619 16:43:47.232166 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 16:43:47.232207 (+ 41us) server_negotiation.cc:407] Connection header received
0619 16:43:47.233682 (+ 1475us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:43:47.233703 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:43:47.234263 (+ 560us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:43:47.234613 (+ 350us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:43:47.235139 (+ 526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:47.236334 (+ 1195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:47.237563 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:47.238238 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:47.238400 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:43:47.239522 (+ 1122us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:43:47.239531 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:43:47.239549 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:43:47.240061 (+ 512us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:43:47.240130 (+ 69us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:43:47.240150 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:43:47.240250 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0619 16:43:47.240454 (+ 204us) server_negotiation.cc:299] Negotiation successful
0619 16:43:47.240741 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":562,"thread_start_us":340,"threads_started":1}
I20260619 16:43:47.242318 5910 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 769504c4735a4b17843633b4a2f8a630: Neither blocks nor log segments found. Creating new log.
I20260619 16:43:47.243319 5910 log.cc:824] T 00000000000000000000000000000000 P 769504c4735a4b17843633b4a2f8a630: Log is configured to *not* fsync() on all Append() calls
I20260619 16:43:47.244964 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 16:43:47.245970 5910 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 769504c4735a4b17843633b4a2f8a630: No bootstrap required, opened a new log
I20260619 16:43:47.253113 5910 raft_consensus.cc:348] T 00000000000000000000000000000000 P 769504c4735a4b17843633b4a2f8a630 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "769504c4735a4b17843633b4a2f8a630" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39395 } }
I20260619 16:43:47.253453 5910 raft_consensus.cc:374] T 00000000000000000000000000000000 P 769504c4735a4b17843633b4a2f8a630 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:43:47.253542 5910 raft_consensus.cc:729] T 00000000000000000000000000000000 P 769504c4735a4b17843633b4a2f8a630 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 769504c4735a4b17843633b4a2f8a630, State: Initialized, Role: FOLLOWER
I20260619 16:43:47.254083 5910 consensus_queue.cc:260] T 00000000000000000000000000000000 P 769504c4735a4b17843633b4a2f8a630 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "769504c4735a4b17843633b4a2f8a630" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39395 } }
I20260619 16:43:47.254308 5910 raft_consensus.cc:388] T 00000000000000000000000000000000 P 769504c4735a4b17843633b4a2f8a630 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:43:47.254426 5910 raft_consensus.cc:482] T 00000000000000000000000000000000 P 769504c4735a4b17843633b4a2f8a630 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:43:47.254568 5910 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 769504c4735a4b17843633b4a2f8a630 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:43:47.256767 5910 raft_consensus.cc:504] T 00000000000000000000000000000000 P 769504c4735a4b17843633b4a2f8a630 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "769504c4735a4b17843633b4a2f8a630" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39395 } }
I20260619 16:43:47.257227 5910 leader_election.cc:302] T 00000000000000000000000000000000 P 769504c4735a4b17843633b4a2f8a630 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 769504c4735a4b17843633b4a2f8a630; no voters:
I20260619 16:43:47.257828 5910 leader_election.cc:288] T 00000000000000000000000000000000 P 769504c4735a4b17843633b4a2f8a630 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 16:43:47.257898 5915 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 769504c4735a4b17843633b4a2f8a630 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:43:47.258534 5915 raft_consensus.cc:686] T 00000000000000000000000000000000 P 769504c4735a4b17843633b4a2f8a630 [term 1 LEADER]: Becoming Leader. State: Replica: 769504c4735a4b17843633b4a2f8a630, State: Running, Role: LEADER
I20260619 16:43:47.259025 5915 consensus_queue.cc:237] T 00000000000000000000000000000000 P 769504c4735a4b17843633b4a2f8a630 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "769504c4735a4b17843633b4a2f8a630" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39395 } }
I20260619 16:43:47.259754 5910 sys_catalog.cc:564] T 00000000000000000000000000000000 P 769504c4735a4b17843633b4a2f8a630 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:43:47.261461 5917 sys_catalog.cc:455] T 00000000000000000000000000000000 P 769504c4735a4b17843633b4a2f8a630 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 769504c4735a4b17843633b4a2f8a630. Latest consensus state: current_term: 1 leader_uuid: "769504c4735a4b17843633b4a2f8a630" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "769504c4735a4b17843633b4a2f8a630" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39395 } } }
I20260619 16:43:47.261857 5917 sys_catalog.cc:458] T 00000000000000000000000000000000 P 769504c4735a4b17843633b4a2f8a630 [sys.catalog]: This master's current role is: LEADER
I20260619 16:43:47.262075 5916 sys_catalog.cc:455] T 00000000000000000000000000000000 P 769504c4735a4b17843633b4a2f8a630 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "769504c4735a4b17843633b4a2f8a630" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "769504c4735a4b17843633b4a2f8a630" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39395 } } }
I20260619 16:43:47.262696 5916 sys_catalog.cc:458] T 00000000000000000000000000000000 P 769504c4735a4b17843633b4a2f8a630 [sys.catalog]: This master's current role is: LEADER
I20260619 16:43:47.264137 5924 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:43:47.268822 5924 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:43:47.277449 5924 catalog_manager.cc:1347] Generated new cluster ID: 09dd49acd16a48be9a1496f3ec9c91dc
I20260619 16:43:47.277562 5924 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:43:47.288654 5924 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 16:43:47.290045 5924 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:43:47.298903 5924 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 769504c4735a4b17843633b4a2f8a630: Generated new TSK 0
I20260619 16:43:47.299646 5924 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 16:43:47.398862 5913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:47.319087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43774 (local address 127.0.97.62:39395)
0619 16:43:47.319269 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0619 16:43:47.319275 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:43:47.319655 (+ 380us) server_negotiation.cc:407] Connection header received
0619 16:43:47.319861 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:43:47.319867 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:43:47.319959 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:43:47.320143 (+ 184us) server_negotiation.cc:226] Negotiated authn=JWT
0619 16:43:47.320556 (+ 413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:47.321649 (+ 1093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:47.322982 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:47.323636 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:47.323756 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:43:47.324229 (+ 473us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0619 16:43:47.398143 (+ 73914us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0619 16:43:47.398566 (+ 423us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:43774: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":59}
W20260619 16:43:47.398988 5939 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:43:47.318929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39395 (local address 127.0.0.1:43774)
0619 16:43:47.319561 (+ 632us) negotiation.cc:107] Waiting for socket to connect
0619 16:43:47.319577 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 16:43:47.319695 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:43:47.320169 (+ 474us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:43:47.320173 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:43:47.320183 (+ 10us) client_negotiation.cc:189] Negotiated authn=JWT
0619 16:43:47.320421 (+ 238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:43:47.320429 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:43:47.321814 (+ 1385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:43:47.321818 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:43:47.322855 (+ 1037us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:43:47.322866 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:43:47.323756 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:43:47.323798 (+ 42us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:43:47.323933 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:43:47.323964 (+ 31us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0619 16:43:47.398454 (+ 74490us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0619 16:43:47.398695 (+ 241us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.97.62:39395: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":491,"thread_start_us":348,"threads_started":1}
W20260619 16:43:47.399001 5913 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:43774: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
I20260619 16:43:47.400585 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu pid 5844
I20260619 16:43:47.401628 5905 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:43:47.498588 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu with pid 5844
2026-06-19T16:43:47Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (737 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (391 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (378 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887385903558-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 16:43:48 dist-test-slave-wvz7 krb5kdc[5954](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 16:43:48 dist-test-slave-wvz7 krb5kdc[5954](info): set up 2 sockets
Jun 19 16:43:48 dist-test-slave-wvz7 krb5kdc[5954](info): commencing operation
krb5kdc: starting...
W20260619 16:43:49.262120 388 mini_kdc.cc:121] Time spent starting KDC: real 0.963s user 0.000s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887385903558-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887385903558-388-0/krb5kdc/test-user.keytab.
Jun 19 16:43:49 dist-test-slave-wvz7 krb5kdc[5954](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887429, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T16:43:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T16:43:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887385903558-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887385903558-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260619 16:43:49.436978 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu
/tmp/dist-test-taskEs2EH8/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887385903558-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887385903558-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887385903558-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887385903558-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37195
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887385903558-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40205
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:37195
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887385903558-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887385903558-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887385903558-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:43:49.741182 5967 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:43:49.741479 5967 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:43:49.741638 5967 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:43:49.749060 5967 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:43:49.749171 5967 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:43:49.749256 5967 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:43:49.749328 5967 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:43:49.749387 5967 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:43:49.759059 5967 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40205
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887385903558-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887385903558-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:37195
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887385903558-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37195
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887385903558-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887385903558-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:43:49.760308 5967 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:43:49.761659 5967 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:43:49.772675 5972 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:43:49.772658 5973 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:43:49.772689 5975 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:43:49.773694 5967 server_base.cc:1034] running on GCE node
I20260619 16:43:49.774505 5967 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:43:49.775887 5967 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:43:49.777081 5967 hybrid_clock.cc:648] HybridClock initialized: now 1781887429777028 us; error 40 us; skew 500 ppm
I20260619 16:43:49.777395 5967 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:43:49 dist-test-slave-wvz7 krb5kdc[5954](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887429, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:43:49.780287 5967 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260619 16:43:49.781714 5967 webserver.cc:460] Webserver started at http://127.0.97.62:33381/ using document root <none> and password file <none>
I20260619 16:43:49.782375 5967 fs_manager.cc:359] Metadata directory not provided
I20260619 16:43:49.782482 5967 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:43:49.782748 5967 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:43:49.785501 5967 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887385903558-388-0/minicluster-data/master-0/data/instance:
uuid: "5a19e6e8f824457d8de5b03cda0ece3f"
format_stamp: "Formatted at 2026-06-19 16:43:49 on dist-test-slave-wvz7"
server_key: "894fe4508a193fe096ed92b461e34a7f"
server_key_iv: "204e35f5e9c4c3a317bc710751c8a6c4"
server_key_version: "encryptionkey@0"
I20260619 16:43:49.786247 5967 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887385903558-388-0/minicluster-data/master-0/wal/instance:
uuid: "5a19e6e8f824457d8de5b03cda0ece3f"
format_stamp: "Formatted at 2026-06-19 16:43:49 on dist-test-slave-wvz7"
server_key: "894fe4508a193fe096ed92b461e34a7f"
server_key_iv: "204e35f5e9c4c3a317bc710751c8a6c4"
server_key_version: "encryptionkey@0"
I20260619 16:43:49.791103 5967 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260619 16:43:49.794281 5982 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:43:49.795585 5967 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260619 16:43:49.795743 5967 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887385903558-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887385903558-388-0/minicluster-data/master-0/wal
uuid: "5a19e6e8f824457d8de5b03cda0ece3f"
format_stamp: "Formatted at 2026-06-19 16:43:49 on dist-test-slave-wvz7"
server_key: "894fe4508a193fe096ed92b461e34a7f"
server_key_iv: "204e35f5e9c4c3a317bc710751c8a6c4"
server_key_version: "encryptionkey@0"
I20260619 16:43:49.795907 5967 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887385903558-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887385903558-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887385903558-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:43:49.818101 5967 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:43:49.822144 5967 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:43:49.822397 5967 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:43:49.838527 5967 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:37195
I20260619 16:43:49.838560 6033 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:37195 every 8 connection(s)
I20260619 16:43:49.839740 5967 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887385903558-388-0/minicluster-data/master-0/data/info.pb
I20260619 16:43:49.842859 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu as pid 5967
I20260619 16:43:49.843068 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887385903558-388-0/minicluster-data/master-0/wal/instance
I20260619 16:43:49.843478 6034 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:43:49.843817 388 external_mini_cluster.cc:1428] Setting key a365ce7aa03315cabcc7b89e4bc96055
I20260619 16:43:49.857401 6034 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5a19e6e8f824457d8de5b03cda0ece3f: Bootstrap starting.
Jun 19 16:43:49 dist-test-slave-wvz7 krb5kdc[5954](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887429, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 16:43:49.860963 6034 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5a19e6e8f824457d8de5b03cda0ece3f: Neither blocks nor log segments found. Creating new log.
I20260619 16:43:49.862251 6034 log.cc:824] T 00000000000000000000000000000000 P 5a19e6e8f824457d8de5b03cda0ece3f: Log is configured to *not* fsync() on all Append() calls
I20260619 16:43:49.865324 6034 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5a19e6e8f824457d8de5b03cda0ece3f: No bootstrap required, opened a new log
I20260619 16:43:49.866421 6037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:49.845673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38132 (local address 127.0.97.62:37195)
0619 16:43:49.846429 (+ 756us) server_negotiation.cc:206] Beginning negotiation
0619 16:43:49.846442 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 16:43:49.846481 (+ 39us) server_negotiation.cc:407] Connection header received
0619 16:43:49.848034 (+ 1553us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:43:49.848055 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:43:49.848427 (+ 372us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:43:49.848743 (+ 316us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:43:49.850402 (+ 1659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:49.851678 (+ 1276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:49.853369 (+ 1691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:49.854106 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:49.854243 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:43:49.858562 (+ 4319us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:43:49.858573 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:43:49.858592 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:43:49.858644 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:43:49.862230 (+ 3586us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:43:49.862885 (+ 655us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:43:49.862893 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:43:49.862901 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:43:49.863064 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:43:49.863459 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:43:49.863463 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:43:49.863466 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:43:49.864123 (+ 657us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:43:49.864375 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0619 16:43:49.864738 (+ 363us) server_negotiation.cc:299] Negotiation successful
0619 16:43:49.865121 (+ 383us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":556,"thread_start_us":307,"threads_started":1}
I20260619 16:43:49.873016 6034 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5a19e6e8f824457d8de5b03cda0ece3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a19e6e8f824457d8de5b03cda0ece3f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37195 } }
I20260619 16:43:49.873350 6034 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5a19e6e8f824457d8de5b03cda0ece3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:43:49.873442 6034 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5a19e6e8f824457d8de5b03cda0ece3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a19e6e8f824457d8de5b03cda0ece3f, State: Initialized, Role: FOLLOWER
I20260619 16:43:49.874008 6034 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5a19e6e8f824457d8de5b03cda0ece3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a19e6e8f824457d8de5b03cda0ece3f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37195 } }
I20260619 16:43:49.874212 6034 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5a19e6e8f824457d8de5b03cda0ece3f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:43:49.874341 6034 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5a19e6e8f824457d8de5b03cda0ece3f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:43:49.874482 6034 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5a19e6e8f824457d8de5b03cda0ece3f [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:43:49.876684 6034 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5a19e6e8f824457d8de5b03cda0ece3f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a19e6e8f824457d8de5b03cda0ece3f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37195 } }
I20260619 16:43:49.877079 6034 leader_election.cc:302] T 00000000000000000000000000000000 P 5a19e6e8f824457d8de5b03cda0ece3f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5a19e6e8f824457d8de5b03cda0ece3f; no voters:
I20260619 16:43:49.877560 6034 leader_election.cc:288] T 00000000000000000000000000000000 P 5a19e6e8f824457d8de5b03cda0ece3f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 16:43:49.877691 6039 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5a19e6e8f824457d8de5b03cda0ece3f [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:43:49.878305 6039 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5a19e6e8f824457d8de5b03cda0ece3f [term 1 LEADER]: Becoming Leader. State: Replica: 5a19e6e8f824457d8de5b03cda0ece3f, State: Running, Role: LEADER
I20260619 16:43:49.878790 6039 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5a19e6e8f824457d8de5b03cda0ece3f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a19e6e8f824457d8de5b03cda0ece3f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37195 } }
I20260619 16:43:49.879377 6034 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5a19e6e8f824457d8de5b03cda0ece3f [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:43:49.881433 6041 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5a19e6e8f824457d8de5b03cda0ece3f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5a19e6e8f824457d8de5b03cda0ece3f. Latest consensus state: current_term: 1 leader_uuid: "5a19e6e8f824457d8de5b03cda0ece3f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a19e6e8f824457d8de5b03cda0ece3f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37195 } } }
I20260619 16:43:49.881862 6041 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5a19e6e8f824457d8de5b03cda0ece3f [sys.catalog]: This master's current role is: LEADER
I20260619 16:43:49.882041 6040 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5a19e6e8f824457d8de5b03cda0ece3f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5a19e6e8f824457d8de5b03cda0ece3f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a19e6e8f824457d8de5b03cda0ece3f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37195 } } }
I20260619 16:43:49.882371 6040 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5a19e6e8f824457d8de5b03cda0ece3f [sys.catalog]: This master's current role is: LEADER
I20260619 16:43:49.883850 6048 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:43:49.888783 6048 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:43:49.897920 6048 catalog_manager.cc:1347] Generated new cluster ID: e3e8c48ca5ad4118bdc9c01e7331486e
I20260619 16:43:49.898015 6048 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:43:49.921702 6048 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 16:43:49.923537 6048 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:43:49.933983 6048 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5a19e6e8f824457d8de5b03cda0ece3f: Generated new TSK 0
I20260619 16:43:49.934880 6048 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887385903558-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887385903558-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260619 16:43:49.982965 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu
/tmp/dist-test-taskEs2EH8/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887385903558-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887385903558-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887385903558-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887385903558-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37195
--keytab_file=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887385903558-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40205
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887385903558-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887385903558-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887385903558-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:43:50.333185 6060 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:43:50.333480 6060 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:43:50.333612 6060 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:43:50.341030 6060 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:43:50.341154 6060 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:43:50.341315 6060 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260619 16:43:50.351161 6060 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40205
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887385903558-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887385903558-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887385903558-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887385903558-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37195
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887385903558-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:43:50.352470 6060 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:43:50.353806 6060 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:43:50.366662 6066 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:43:50.366706 6065 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:43:50.366775 6068 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:43:50.367515 6060 server_base.cc:1034] running on GCE node
I20260619 16:43:50.368105 6060 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:43:50.369032 6060 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:43:50.370249 6060 hybrid_clock.cc:648] HybridClock initialized: now 1781887430370191 us; error 47 us; skew 500 ppm
I20260619 16:43:50.370563 6060 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:43:50 dist-test-slave-wvz7 krb5kdc[5954](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887430, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:43:50.373844 6060 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260619 16:43:50.375281 6060 webserver.cc:460] Webserver started at http://127.0.97.1:36433/ using document root <none> and password file <none>
I20260619 16:43:50.375957 6060 fs_manager.cc:359] Metadata directory not provided
I20260619 16:43:50.376068 6060 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:43:50.376346 6060 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:43:50.379182 6060 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887385903558-388-0/minicluster-data/ts-0/data/instance:
uuid: "c3d66f9bbb6d4312b8de4db18fb7ba93"
format_stamp: "Formatted at 2026-06-19 16:43:50 on dist-test-slave-wvz7"
server_key: "bb172db7b38333f1afb293fecbcf3ecc"
server_key_iv: "d6424f53a61af1de4fc5626063712a52"
server_key_version: "encryptionkey@0"
I20260619 16:43:50.379954 6060 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887385903558-388-0/minicluster-data/ts-0/wal/instance:
uuid: "c3d66f9bbb6d4312b8de4db18fb7ba93"
format_stamp: "Formatted at 2026-06-19 16:43:50 on dist-test-slave-wvz7"
server_key: "bb172db7b38333f1afb293fecbcf3ecc"
server_key_iv: "d6424f53a61af1de4fc5626063712a52"
server_key_version: "encryptionkey@0"
I20260619 16:43:50.384986 6060 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260619 16:43:50.389034 6075 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:43:50.390394 6060 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 16:43:50.390535 6060 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887385903558-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887385903558-388-0/minicluster-data/ts-0/wal
uuid: "c3d66f9bbb6d4312b8de4db18fb7ba93"
format_stamp: "Formatted at 2026-06-19 16:43:50 on dist-test-slave-wvz7"
server_key: "bb172db7b38333f1afb293fecbcf3ecc"
server_key_iv: "d6424f53a61af1de4fc5626063712a52"
server_key_version: "encryptionkey@0"
I20260619 16:43:50.390725 6060 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887385903558-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887385903558-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887385903558-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:43:50.411960 6060 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:43:50.416435 6060 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:43:50.416744 6060 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:43:50.417783 6060 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:43:50.419145 6060 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:43:50.419232 6060 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:43:50.419363 6060 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:43:50.419420 6060 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:43:50.450054 6060 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:43485
I20260619 16:43:50.450083 6187 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:43485 every 8 connection(s)
I20260619 16:43:50.451375 6060 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887385903558-388-0/minicluster-data/ts-0/data/info.pb
I20260619 16:43:50.454650 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu as pid 6060
I20260619 16:43:50.454850 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887385903558-388-0/minicluster-data/ts-0/wal/instance
I20260619 16:43:50.455638 388 external_mini_cluster.cc:1428] Setting key 913d079d99a919db8598b9d4e1e514e6
Jun 19 16:43:50 dist-test-slave-wvz7 krb5kdc[5954](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887430, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 16:43:50.472321 6190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:50.453859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:51801 (local address 127.0.97.62:37195)
0619 16:43:50.454326 (+ 467us) server_negotiation.cc:206] Beginning negotiation
0619 16:43:50.454333 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:43:50.455389 (+ 1056us) server_negotiation.cc:407] Connection header received
0619 16:43:50.457451 (+ 2062us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:43:50.457456 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:43:50.457529 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:43:50.457617 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:43:50.459500 (+ 1883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:50.460184 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:50.462218 (+ 2034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:50.462686 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:50.462786 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:43:50.466591 (+ 3805us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:43:50.466597 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:43:50.466601 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:43:50.466638 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:43:50.469429 (+ 2791us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:43:50.470129 (+ 700us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:43:50.470134 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:43:50.470138 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:43:50.470274 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:43:50.470622 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:43:50.470626 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:43:50.470629 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:43:50.470935 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:43:50.471078 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0619 16:43:50.471717 (+ 639us) server_negotiation.cc:299] Negotiation successful
0619 16:43:50.471942 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":315,"thread_start_us":206,"threads_started":1}
I20260619 16:43:50.473161 6191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:50.454384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37195 (local address 127.0.97.1:51801)
0619 16:43:50.455202 (+ 818us) negotiation.cc:107] Waiting for socket to connect
0619 16:43:50.455258 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0619 16:43:50.457177 (+ 1919us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:43:50.457911 (+ 734us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:43:50.457922 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:43:50.458570 (+ 648us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:43:50.459331 (+ 761us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:43:50.459354 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:43:50.460318 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:43:50.460323 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:43:50.462088 (+ 1765us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:43:50.462101 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:43:50.462891 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:43:50.462897 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:43:50.463153 (+ 256us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:43:50.463869 (+ 716us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:43:50.463903 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:43:50.466354 (+ 2451us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:43:50.469548 (+ 3194us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:43:50.469554 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:43:50.469567 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:43:50.470018 (+ 451us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:43:50.470362 (+ 344us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:43:50.470365 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:43:50.470367 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:43:50.470529 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:43:50.471041 (+ 512us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:43:50.471047 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:43:50.471367 (+ 320us) client_negotiation.cc:769] Sending connection context
0619 16:43:50.471581 (+ 214us) client_negotiation.cc:240] Negotiation successful
0619 16:43:50.472057 (+ 476us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":533,"thread_start_us":323,"threads_started":1}
I20260619 16:43:50.474692 6188 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37195
I20260619 16:43:50.474978 6188 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260619 16:43:50.475703 6188 heartbeater.cc:511] Master 127.0.97.62:37195 requested a full tablet report, sending...
I20260619 16:43:50.477792 5999 ts_manager.cc:194] Registered new tserver with Master: c3d66f9bbb6d4312b8de4db18fb7ba93 (127.0.97.1:43485)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260619 16:43:50.479280 5999 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:51801
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887385903558-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887385903558-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260619 16:43:50.501109 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu
/tmp/dist-test-taskEs2EH8/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887385903558-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887385903558-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887385903558-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887385903558-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37195
--keytab_file=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887385903558-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40205
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887385903558-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887385903558-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887385903558-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:43:50.810256 6194 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:43:50.810566 6194 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:43:50.810717 6194 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:43:50.817994 6194 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:43:50.818121 6194 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:43:50.818270 6194 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260619 16:43:50.828224 6194 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40205
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887385903558-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887385903558-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887385903558-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887385903558-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37195
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887385903558-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:43:50.829530 6194 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:43:50.830863 6194 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:43:50.845134 6200 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:43:50.845718 6202 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:43:50.845228 6199 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:43:50.847312 6194 server_base.cc:1034] running on GCE node
I20260619 16:43:50.847983 6194 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:43:50.848973 6194 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:43:50.850179 6194 hybrid_clock.cc:648] HybridClock initialized: now 1781887430850107 us; error 64 us; skew 500 ppm
I20260619 16:43:50.850472 6194 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:43:50 dist-test-slave-wvz7 krb5kdc[5954](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887430, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:43:50.853541 6194 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260619 16:43:50.855028 6194 webserver.cc:460] Webserver started at http://127.0.97.2:32875/ using document root <none> and password file <none>
I20260619 16:43:50.855700 6194 fs_manager.cc:359] Metadata directory not provided
I20260619 16:43:50.855811 6194 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:43:50.856074 6194 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:43:50.859016 6194 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887385903558-388-0/minicluster-data/ts-1/data/instance:
uuid: "45fdfb60879d4b6ebef5a509b10b6d0b"
format_stamp: "Formatted at 2026-06-19 16:43:50 on dist-test-slave-wvz7"
server_key: "ff22783619ba53fa10fb49737dcde329"
server_key_iv: "79f2a2c5ad024fdc1186b0fe0e5c6fa9"
server_key_version: "encryptionkey@0"
I20260619 16:43:50.859748 6194 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887385903558-388-0/minicluster-data/ts-1/wal/instance:
uuid: "45fdfb60879d4b6ebef5a509b10b6d0b"
format_stamp: "Formatted at 2026-06-19 16:43:50 on dist-test-slave-wvz7"
server_key: "ff22783619ba53fa10fb49737dcde329"
server_key_iv: "79f2a2c5ad024fdc1186b0fe0e5c6fa9"
server_key_version: "encryptionkey@0"
I20260619 16:43:50.864686 6194 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.003s
I20260619 16:43:50.867956 6209 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:43:50.869234 6194 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260619 16:43:50.869378 6194 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887385903558-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887385903558-388-0/minicluster-data/ts-1/wal
uuid: "45fdfb60879d4b6ebef5a509b10b6d0b"
format_stamp: "Formatted at 2026-06-19 16:43:50 on dist-test-slave-wvz7"
server_key: "ff22783619ba53fa10fb49737dcde329"
server_key_iv: "79f2a2c5ad024fdc1186b0fe0e5c6fa9"
server_key_version: "encryptionkey@0"
I20260619 16:43:50.869557 6194 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887385903558-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887385903558-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887385903558-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:43:50.883662 6194 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:43:50.887634 6194 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:43:50.887895 6194 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:43:50.888793 6194 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:43:50.890038 6194 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:43:50.890107 6194 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:43:50.890192 6194 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:43:50.890235 6194 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:43:50.919482 6194 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:44303
I20260619 16:43:50.919523 6321 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:44303 every 8 connection(s)
I20260619 16:43:50.920696 6194 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887385903558-388-0/minicluster-data/ts-1/data/info.pb
I20260619 16:43:50.929016 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu as pid 6194
I20260619 16:43:50.929198 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887385903558-388-0/minicluster-data/ts-1/wal/instance
I20260619 16:43:50.929770 388 external_mini_cluster.cc:1428] Setting key d508521c339079d03ad1635957e7c903
Jun 19 16:43:50 dist-test-slave-wvz7 krb5kdc[5954](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887430, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 16:43:50.940017 6190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:50.923142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:56991 (local address 127.0.97.62:37195)
0619 16:43:50.923302 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0619 16:43:50.923306 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:43:50.924367 (+ 1061us) server_negotiation.cc:407] Connection header received
0619 16:43:50.926132 (+ 1765us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:43:50.926137 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:43:50.926212 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:43:50.926299 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:43:50.927768 (+ 1469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:50.928454 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:50.930627 (+ 2173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:50.931062 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:50.931170 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:43:50.934896 (+ 3726us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:43:50.934901 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:43:50.934904 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:43:50.934936 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:43:50.937163 (+ 2227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:43:50.937880 (+ 717us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:43:50.937884 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:43:50.937886 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:43:50.938005 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:43:50.938413 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:43:50.938416 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:43:50.938418 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:43:50.938670 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:43:50.938794 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0619 16:43:50.939475 (+ 681us) server_negotiation.cc:299] Negotiation successful
0619 16:43:50.939693 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260619 16:43:50.940999 6324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:50.923533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37195 (local address 127.0.97.2:56991)
0619 16:43:50.924196 (+ 663us) negotiation.cc:107] Waiting for socket to connect
0619 16:43:50.924242 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0619 16:43:50.925865 (+ 1623us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:43:50.926508 (+ 643us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:43:50.926517 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:43:50.927046 (+ 529us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:43:50.927603 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:43:50.927619 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:43:50.928560 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:43:50.928564 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:43:50.930483 (+ 1919us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:43:50.930494 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:43:50.931276 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:43:50.931283 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:43:50.931491 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:43:50.932145 (+ 654us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:43:50.932174 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:43:50.934678 (+ 2504us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:43:50.937299 (+ 2621us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:43:50.937309 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:43:50.937326 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:43:50.937728 (+ 402us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:43:50.938107 (+ 379us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:43:50.938112 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:43:50.938116 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:43:50.938302 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:43:50.938787 (+ 485us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:43:50.938795 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:43:50.939121 (+ 326us) client_negotiation.cc:769] Sending connection context
0619 16:43:50.939350 (+ 229us) client_negotiation.cc:240] Negotiation successful
0619 16:43:50.939896 (+ 546us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":451,"thread_start_us":270,"threads_started":1}
I20260619 16:43:50.942489 6322 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37195
I20260619 16:43:50.942777 6322 heartbeater.cc:464] Registering TS with master...
I20260619 16:43:50.943492 6322 heartbeater.cc:511] Master 127.0.97.62:37195 requested a full tablet report, sending...
I20260619 16:43:50.944976 5999 ts_manager.cc:194] Registered new tserver with Master: 45fdfb60879d4b6ebef5a509b10b6d0b (127.0.97.2:44303)
I20260619 16:43:50.945772 5999 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:56991
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887385903558-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887385903558-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260619 16:43:50.970471 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu
/tmp/dist-test-taskEs2EH8/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887385903558-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887385903558-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887385903558-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887385903558-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37195
--keytab_file=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887385903558-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40205
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887385903558-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887385903558-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887385903558-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:43:51.277918 6327 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:43:51.278183 6327 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:43:51.278306 6327 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:43:51.285784 6327 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:43:51.285889 6327 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:43:51.286037 6327 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260619 16:43:51.295621 6327 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40205
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887385903558-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887385903558-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887385903558-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887385903558-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37195
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887385903558-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:43:51.296820 6327 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:43:51.298079 6327 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:43:51.310026 6333 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:43:51.310107 6335 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:43:51.310065 6332 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:43:51.311056 6327 server_base.cc:1034] running on GCE node
I20260619 16:43:51.311621 6327 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:43:51.312460 6327 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:43:51.313658 6327 hybrid_clock.cc:648] HybridClock initialized: now 1781887431313610 us; error 41 us; skew 500 ppm
I20260619 16:43:51.313968 6327 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:43:51 dist-test-slave-wvz7 krb5kdc[5954](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887431, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:43:51.316805 6327 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260619 16:43:51.318107 6327 webserver.cc:460] Webserver started at http://127.0.97.3:42225/ using document root <none> and password file <none>
I20260619 16:43:51.318702 6327 fs_manager.cc:359] Metadata directory not provided
I20260619 16:43:51.318792 6327 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:43:51.319017 6327 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:43:51.321692 6327 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887385903558-388-0/minicluster-data/ts-2/data/instance:
uuid: "7d38abf5b62743f49fb4aae47753120c"
format_stamp: "Formatted at 2026-06-19 16:43:51 on dist-test-slave-wvz7"
server_key: "a301d524d6f27f8c997b7a41194b9365"
server_key_iv: "1752d73bef7a97d55e33bc92d4246a5d"
server_key_version: "encryptionkey@0"
I20260619 16:43:51.322422 6327 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887385903558-388-0/minicluster-data/ts-2/wal/instance:
uuid: "7d38abf5b62743f49fb4aae47753120c"
format_stamp: "Formatted at 2026-06-19 16:43:51 on dist-test-slave-wvz7"
server_key: "a301d524d6f27f8c997b7a41194b9365"
server_key_iv: "1752d73bef7a97d55e33bc92d4246a5d"
server_key_version: "encryptionkey@0"
I20260619 16:43:51.327036 6327 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260619 16:43:51.330191 6342 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:43:51.331458 6327 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260619 16:43:51.331593 6327 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887385903558-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887385903558-388-0/minicluster-data/ts-2/wal
uuid: "7d38abf5b62743f49fb4aae47753120c"
format_stamp: "Formatted at 2026-06-19 16:43:51 on dist-test-slave-wvz7"
server_key: "a301d524d6f27f8c997b7a41194b9365"
server_key_iv: "1752d73bef7a97d55e33bc92d4246a5d"
server_key_version: "encryptionkey@0"
I20260619 16:43:51.331760 6327 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887385903558-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887385903558-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887385903558-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:43:51.345302 6327 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:43:51.349362 6327 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:43:51.349637 6327 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:43:51.350602 6327 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:43:51.351898 6327 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:43:51.351975 6327 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:43:51.352072 6327 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:43:51.352123 6327 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:43:51.381098 6327 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:45181
I20260619 16:43:51.381151 6454 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:45181 every 8 connection(s)
I20260619 16:43:51.382503 6327 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887385903558-388-0/minicluster-data/ts-2/data/info.pb
I20260619 16:43:51.387982 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu as pid 6327
I20260619 16:43:51.388145 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887385903558-388-0/minicluster-data/ts-2/wal/instance
I20260619 16:43:51.388594 388 external_mini_cluster.cc:1428] Setting key 892bff0efcd855a6b351506b3361b94f
Jun 19 16:43:51 dist-test-slave-wvz7 krb5kdc[5954](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887431, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 16:43:51.402263 6190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:51.384974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:39449 (local address 127.0.97.62:37195)
0619 16:43:51.385124 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0619 16:43:51.385129 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:43:51.386326 (+ 1197us) server_negotiation.cc:407] Connection header received
0619 16:43:51.388044 (+ 1718us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:43:51.388050 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:43:51.388146 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:43:51.388254 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:43:51.390137 (+ 1883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:51.391251 (+ 1114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:51.393079 (+ 1828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:51.393457 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:51.393555 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:43:51.396808 (+ 3253us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:43:51.396813 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:43:51.396817 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:43:51.396849 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:43:51.399397 (+ 2548us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:43:51.400040 (+ 643us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:43:51.400043 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:43:51.400045 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:43:51.400165 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:43:51.400646 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:43:51.400650 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:43:51.400653 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:43:51.400944 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:43:51.401071 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0619 16:43:51.401823 (+ 752us) server_negotiation.cc:299] Negotiation successful
0619 16:43:51.402023 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260619 16:43:51.403324 6457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:51.385415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37195 (local address 127.0.97.3:39449)
0619 16:43:51.386165 (+ 750us) negotiation.cc:107] Waiting for socket to connect
0619 16:43:51.386210 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0619 16:43:51.387782 (+ 1572us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:43:51.388471 (+ 689us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:43:51.388479 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:43:51.389118 (+ 639us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:43:51.389923 (+ 805us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:43:51.389949 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:43:51.391424 (+ 1475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:43:51.391430 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:43:51.392957 (+ 1527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:43:51.392969 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:43:51.393616 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:43:51.393620 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:43:51.393884 (+ 264us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:43:51.394466 (+ 582us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:43:51.394485 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:43:51.396607 (+ 2122us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:43:51.399532 (+ 2925us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:43:51.399539 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:43:51.399551 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:43:51.399947 (+ 396us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:43:51.400296 (+ 349us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:43:51.400301 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:43:51.400305 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:43:51.400497 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:43:51.401081 (+ 584us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:43:51.401087 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:43:51.401412 (+ 325us) client_negotiation.cc:769] Sending connection context
0619 16:43:51.401640 (+ 228us) client_negotiation.cc:240] Negotiation successful
0619 16:43:51.402163 (+ 523us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":516,"thread_start_us":346,"threads_started":1}
I20260619 16:43:51.404654 6455 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37195
I20260619 16:43:51.404938 6455 heartbeater.cc:464] Registering TS with master...
I20260619 16:43:51.405627 6455 heartbeater.cc:511] Master 127.0.97.62:37195 requested a full tablet report, sending...
I20260619 16:43:51.406925 5998 ts_manager.cc:194] Registered new tserver with Master: 7d38abf5b62743f49fb4aae47753120c (127.0.97.3:45181)
I20260619 16:43:51.407750 5998 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:39449
I20260619 16:43:51.416901 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887385903558-388-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887385903558-388-0/krb5kdc/test-admin.keytab.
Jun 19 16:43:51 dist-test-slave-wvz7 krb5kdc[5954](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887431, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:43:51.440630 388 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260619 16:43:51.445142 6462 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:43:51.443033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43998 (local address 127.0.97.1:43485)
0619 16:43:51.443659 (+ 626us) server_negotiation.cc:206] Beginning negotiation
0619 16:43:51.443667 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 16:43:51.443694 (+ 27us) server_negotiation.cc:407] Connection header received
0619 16:43:51.443776 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:43:51.443783 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:43:51.443989 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:43:51.444117 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:43:51.444851 (+ 734us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43998: BlockingRecv error: recv got EOF from 127.0.0.1:43998 (error 108)
Metrics: {"server-negotiator.queue_time_us":472,"thread_start_us":375,"threads_started":1}
W20260619 16:43:51.445299 388 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887385903558-388-0/krb5kdc/krb5cc' of type FILE
Jun 19 16:43:51 dist-test-slave-wvz7 krb5kdc[5954](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887431, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:43:51.446760 388 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 19 16:43:51 dist-test-slave-wvz7 krb5kdc[5954](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887431, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:43:51.448074 388 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 19 16:43:51 dist-test-slave-wvz7 krb5kdc[5954](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887431, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260619 16:43:51.462987 6462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:51.449658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44010 (local address 127.0.97.1:43485)
0619 16:43:51.449856 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0619 16:43:51.449861 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:43:51.450069 (+ 208us) server_negotiation.cc:407] Connection header received
0619 16:43:51.450219 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:43:51.450224 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:43:51.450301 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:43:51.450410 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:43:51.451371 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:51.452858 (+ 1487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:51.454155 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:51.454941 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:51.455063 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:43:51.458444 (+ 3381us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:43:51.458454 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:43:51.458467 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:43:51.458500 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:43:51.460795 (+ 2295us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:43:51.461352 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:43:51.461357 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:43:51.461360 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:43:51.461471 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:43:51.461789 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:43:51.461792 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:43:51.461794 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:43:51.462095 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:43:51.462312 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0619 16:43:51.462560 (+ 248us) server_negotiation.cc:299] Negotiation successful
0619 16:43:51.462743 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
W20260619 16:43:51.468382 6462 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:43:51.466759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44018 (local address 127.0.97.1:43485)
0619 16:43:51.466937 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0619 16:43:51.466942 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:43:51.467241 (+ 299us) server_negotiation.cc:407] Connection header received
0619 16:43:51.467384 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:43:51.467389 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:43:51.467450 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:43:51.467523 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:43:51.468178 (+ 655us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44018: BlockingRecv error: recv got EOF from 127.0.0.1:44018 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
W20260619 16:43:51.468516 388 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887385903558-388-0/krb5kdc/krb5cc' of type FILE
Jun 19 16:43:51 dist-test-slave-wvz7 krb5kdc[5954](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887431, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:43:51.470033 388 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 19 16:43:51 dist-test-slave-wvz7 krb5kdc[5954](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887431, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:43:51.471349 388 init.cc:295] Successfully reacquired a new kerberos TGT
I20260619 16:43:51.482594 6188 heartbeater.cc:503] Master 127.0.97.62:37195 was elected leader, sending a full tablet report...
Jun 19 16:43:51 dist-test-slave-wvz7 krb5kdc[5954](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887431, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260619 16:43:51.488440 6462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:51.472558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44026 (local address 127.0.97.1:43485)
0619 16:43:51.472813 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0619 16:43:51.472818 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:43:51.472964 (+ 146us) server_negotiation.cc:407] Connection header received
0619 16:43:51.473114 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:43:51.473119 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:43:51.473180 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:43:51.473262 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:43:51.474198 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:51.474881 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:51.476000 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:51.476433 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:51.476534 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:43:51.483732 (+ 7198us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:43:51.483739 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:43:51.483743 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:43:51.483794 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:43:51.486318 (+ 2524us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:43:51.486948 (+ 630us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:43:51.486952 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:43:51.486954 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:43:51.487068 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:43:51.487338 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:43:51.487341 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:43:51.487343 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:43:51.487605 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:43:51.487775 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0619 16:43:51.487984 (+ 209us) server_negotiation.cc:299] Negotiation successful
0619 16:43:51.488139 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126}
W20260619 16:43:51.492695 6462 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:43:51.490964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44036 (local address 127.0.97.1:43485)
0619 16:43:51.491166 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0619 16:43:51.491171 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:43:51.491377 (+ 206us) server_negotiation.cc:407] Connection header received
0619 16:43:51.491543 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:43:51.491547 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:43:51.491626 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:43:51.491711 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:43:51.492494 (+ 783us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44036: BlockingRecv error: recv got EOF from 127.0.0.1:44036 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
W20260619 16:43:51.492937 388 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887385903558-388-0/krb5kdc/krb5cc' of type FILE
Jun 19 16:43:51 dist-test-slave-wvz7 krb5kdc[5954](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887431, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:43:51.494573 388 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 19 16:43:51 dist-test-slave-wvz7 krb5kdc[5954](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887431, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:43:51.495857 388 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 19 16:43:51 dist-test-slave-wvz7 krb5kdc[5954](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887431, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260619 16:43:51.508745 6462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:51.497053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44038 (local address 127.0.97.1:43485)
0619 16:43:51.497250 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0619 16:43:51.497256 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:43:51.497443 (+ 187us) server_negotiation.cc:407] Connection header received
0619 16:43:51.497559 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:43:51.497563 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:43:51.497619 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:43:51.497716 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:43:51.498653 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:51.499394 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:51.500677 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:51.501065 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:51.501158 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:43:51.504249 (+ 3091us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:43:51.504254 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:43:51.504258 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:43:51.504310 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:43:51.506715 (+ 2405us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:43:51.507238 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:43:51.507242 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:43:51.507244 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:43:51.507359 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:43:51.507633 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:43:51.507635 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:43:51.507637 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:43:51.507889 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:43:51.508093 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0619 16:43:51.508264 (+ 171us) server_negotiation.cc:299] Negotiation successful
0619 16:43:51.508435 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260619 16:43:51.510273 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu pid 6060
I20260619 16:43:51.520737 6462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:51.510866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44048 (local address 127.0.97.1:43485)
0619 16:43:51.511041 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0619 16:43:51.511046 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:43:51.511278 (+ 232us) server_negotiation.cc:407] Connection header received
0619 16:43:51.511384 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:43:51.511387 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:43:51.511437 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:43:51.511506 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:43:51.512309 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:51.513008 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:51.514197 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:51.514625 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:51.514715 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:43:51.516410 (+ 1695us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:43:51.516414 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:43:51.516417 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:43:51.516445 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:43:51.518766 (+ 2321us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:43:51.519286 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:43:51.519289 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:43:51.519292 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:43:51.519397 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:43:51.519647 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:43:51.519650 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:43:51.519652 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:43:51.519886 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:43:51.520063 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0619 16:43:51.520284 (+ 221us) server_negotiation.cc:299] Negotiation successful
0619 16:43:51.520429 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260619 16:43:51.521222 6183 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:43:51.601982 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu with pid 6060
I20260619 16:43:51.614141 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu pid 6194
Jun 19 16:43:51 dist-test-slave-wvz7 krb5kdc[5954](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887431, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260619 16:43:51.628443 6475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:51.615013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42562 (local address 127.0.97.2:44303)
0619 16:43:51.615493 (+ 480us) server_negotiation.cc:206] Beginning negotiation
0619 16:43:51.615500 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:43:51.615522 (+ 22us) server_negotiation.cc:407] Connection header received
0619 16:43:51.615579 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:43:51.615588 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:43:51.615794 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:43:51.615936 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:43:51.616867 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:51.617896 (+ 1029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:51.619072 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:51.619609 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:51.619714 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:43:51.623280 (+ 3566us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:43:51.623289 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:43:51.623303 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:43:51.623334 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:43:51.625906 (+ 2572us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:43:51.626519 (+ 613us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:43:51.626528 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:43:51.626532 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:43:51.626647 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:43:51.626983 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:43:51.626988 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:43:51.626992 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:43:51.627310 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:43:51.627581 (+ 271us) server_negotiation.cc:1035] Waiting for connection context
0619 16:43:51.627915 (+ 334us) server_negotiation.cc:299] Negotiation successful
0619 16:43:51.628143 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":361,"thread_start_us":235,"threads_started":1}
I20260619 16:43:51.629629 6317 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:43:51.708577 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu with pid 6194
I20260619 16:43:51.720203 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu pid 6327
Jun 19 16:43:51 dist-test-slave-wvz7 krb5kdc[5954](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887431, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260619 16:43:51.734869 6477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:51.721248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42980 (local address 127.0.97.3:45181)
0619 16:43:51.721982 (+ 734us) server_negotiation.cc:206] Beginning negotiation
0619 16:43:51.721989 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:43:51.722021 (+ 32us) server_negotiation.cc:407] Connection header received
0619 16:43:51.722088 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:43:51.722095 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:43:51.722304 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:43:51.722442 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:43:51.723389 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:51.724474 (+ 1085us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:51.725621 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:51.726266 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:51.726371 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:43:51.730096 (+ 3725us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:43:51.730107 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:43:51.730127 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:43:51.730160 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:43:51.732538 (+ 2378us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:43:51.733123 (+ 585us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:43:51.733130 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:43:51.733132 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:43:51.733238 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:43:51.733527 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:43:51.733529 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:43:51.733531 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:43:51.733899 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:43:51.734124 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0619 16:43:51.734392 (+ 268us) server_negotiation.cc:299] Negotiation successful
0619 16:43:51.734575 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":584,"thread_start_us":370,"threads_started":1}
I20260619 16:43:51.736039 6450 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:43:51.815506 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu with pid 6327
I20260619 16:43:51.827415 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu pid 5967
I20260619 16:43:51.828472 6029 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:43:51.928471 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu with pid 5967
2026-06-19T16:43:51Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (3665 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887385903558-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 16:43:52 dist-test-slave-wvz7 krb5kdc[6482](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 16:43:52 dist-test-slave-wvz7 krb5kdc[6482](info): set up 2 sockets
Jun 19 16:43:52 dist-test-slave-wvz7 krb5kdc[6482](info): commencing operation
krb5kdc: starting...
W20260619 16:43:52.922087 388 mini_kdc.cc:121] Time spent starting KDC: real 0.958s user 0.007s sys 0.010s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887385903558-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887385903558-388-0/krb5kdc/test-user.keytab.
Jun 19 16:43:53 dist-test-slave-wvz7 krb5kdc[6482](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887433, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T16:43:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T16:43:53Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887385903558-388-0/krb5kdc/oryx_127.0.97.62.keytab.
Entry for principal oryx/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887385903558-388-0/krb5kdc/oryx_127.0.97.62.keytab.
I20260619 16:43:53.105588 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu
/tmp/dist-test-taskEs2EH8/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887385903558-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887385903558-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887385903558-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887385903558-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:46817
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887385903558-388-0/krb5kdc/oryx_127.0.97.62.keytab
--principal=oryx/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44537
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:46817
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887385903558-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887385903558-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887385903558-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:43:53.413805 6495 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:43:53.414083 6495 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:43:53.414203 6495 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:43:53.421484 6495 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:43:53.421586 6495 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:43:53.421677 6495 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:43:53.421765 6495 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:43:53.421826 6495 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:43:53.431782 6495 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44537
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887385903558-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887385903558-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:46817
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887385903558-388-0/krb5kdc/oryx_127.0.97.62.keytab
--principal=oryx/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:46817
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887385903558-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887385903558-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:43:53.433024 6495 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:43:53.434365 6495 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:43:53.444983 6503 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:43:53.444988 6501 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:43:53.445353 6500 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:43:53.445688 6495 server_base.cc:1034] running on GCE node
I20260619 16:43:53.446455 6495 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:43:53.447854 6495 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:43:53.449092 6495 hybrid_clock.cc:648] HybridClock initialized: now 1781887433449038 us; error 51 us; skew 500 ppm
I20260619 16:43:53.449419 6495 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:43:53 dist-test-slave-wvz7 krb5kdc[6482](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887433, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:43:53.452436 6495 init.cc:377] Logged in from keytab as oryx/127.0.97.62@KRBTEST.COM (short username oryx)
I20260619 16:43:53.453948 6495 webserver.cc:460] Webserver started at http://127.0.97.62:37501/ using document root <none> and password file <none>
I20260619 16:43:53.454627 6495 fs_manager.cc:359] Metadata directory not provided
I20260619 16:43:53.454749 6495 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:43:53.455021 6495 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:43:53.457984 6495 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887385903558-388-0/minicluster-data/master-0/data/instance:
uuid: "5ba5d12f1093420e96661d1c478add9b"
format_stamp: "Formatted at 2026-06-19 16:43:53 on dist-test-slave-wvz7"
server_key: "ed79cdb84a97a4a776c41c2b89d16a1b"
server_key_iv: "2d5d3ccac36d2d05a11493122cdc5f91"
server_key_version: "encryptionkey@0"
I20260619 16:43:53.458707 6495 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887385903558-388-0/minicluster-data/master-0/wal/instance:
uuid: "5ba5d12f1093420e96661d1c478add9b"
format_stamp: "Formatted at 2026-06-19 16:43:53 on dist-test-slave-wvz7"
server_key: "ed79cdb84a97a4a776c41c2b89d16a1b"
server_key_iv: "2d5d3ccac36d2d05a11493122cdc5f91"
server_key_version: "encryptionkey@0"
I20260619 16:43:53.463786 6495 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 16:43:53.467132 6510 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:43:53.468667 6495 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260619 16:43:53.468839 6495 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887385903558-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887385903558-388-0/minicluster-data/master-0/wal
uuid: "5ba5d12f1093420e96661d1c478add9b"
format_stamp: "Formatted at 2026-06-19 16:43:53 on dist-test-slave-wvz7"
server_key: "ed79cdb84a97a4a776c41c2b89d16a1b"
server_key_iv: "2d5d3ccac36d2d05a11493122cdc5f91"
server_key_version: "encryptionkey@0"
I20260619 16:43:53.469017 6495 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887385903558-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887385903558-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887385903558-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:43:53.482774 6495 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:43:53.486944 6495 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:43:53.487164 6495 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:43:53.502707 6495 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:46817
I20260619 16:43:53.502753 6561 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:46817 every 8 connection(s)
I20260619 16:43:53.503983 6495 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887385903558-388-0/minicluster-data/master-0/data/info.pb
I20260619 16:43:53.507671 6562 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:43:53.512393 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu as pid 6495
I20260619 16:43:53.512566 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887385903558-388-0/minicluster-data/master-0/wal/instance
I20260619 16:43:53.513149 388 external_mini_cluster.cc:1428] Setting key c753e79260bd8e8d5cee3601a3fb4031
I20260619 16:43:53.517868 6562 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5ba5d12f1093420e96661d1c478add9b: Bootstrap starting.
I20260619 16:43:53.521377 6562 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5ba5d12f1093420e96661d1c478add9b: Neither blocks nor log segments found. Creating new log.
I20260619 16:43:53.522637 6562 log.cc:824] T 00000000000000000000000000000000 P 5ba5d12f1093420e96661d1c478add9b: Log is configured to *not* fsync() on all Append() calls
I20260619 16:43:53.526052 6562 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5ba5d12f1093420e96661d1c478add9b: No bootstrap required, opened a new log
Jun 19 16:43:53 dist-test-slave-wvz7 krb5kdc[6482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887433, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260619 16:43:53.533600 6562 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5ba5d12f1093420e96661d1c478add9b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ba5d12f1093420e96661d1c478add9b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46817 } }
I20260619 16:43:53.534008 6562 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5ba5d12f1093420e96661d1c478add9b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:43:53.534118 6562 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5ba5d12f1093420e96661d1c478add9b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ba5d12f1093420e96661d1c478add9b, State: Initialized, Role: FOLLOWER
I20260619 16:43:53.534914 6562 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5ba5d12f1093420e96661d1c478add9b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ba5d12f1093420e96661d1c478add9b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46817 } }
I20260619 16:43:53.535135 6562 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5ba5d12f1093420e96661d1c478add9b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:43:53.535276 6562 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5ba5d12f1093420e96661d1c478add9b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:43:53.535425 6562 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5ba5d12f1093420e96661d1c478add9b [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:43:53.535950 6565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:53.515032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46036 (local address 127.0.97.62:46817)
0619 16:43:53.515806 (+ 774us) server_negotiation.cc:206] Beginning negotiation
0619 16:43:53.515820 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 16:43:53.515859 (+ 39us) server_negotiation.cc:407] Connection header received
0619 16:43:53.517466 (+ 1607us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:43:53.517491 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:43:53.517921 (+ 430us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:43:53.518264 (+ 343us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:43:53.519991 (+ 1727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:53.521264 (+ 1273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:53.522956 (+ 1692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:53.523999 (+ 1043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:53.524130 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:43:53.528556 (+ 4426us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:43:53.528564 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:43:53.528583 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:43:53.528620 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:43:53.531635 (+ 3015us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:43:53.532277 (+ 642us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:43:53.532287 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:43:53.532293 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:43:53.532466 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:43:53.532846 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:43:53.532849 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:43:53.532851 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:43:53.533414 (+ 563us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:43:53.533681 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0619 16:43:53.534172 (+ 491us) server_negotiation.cc:299] Negotiation successful
0619 16:43:53.534795 (+ 623us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":574,"thread_start_us":364,"threads_started":1}
I20260619 16:43:53.538087 6562 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5ba5d12f1093420e96661d1c478add9b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ba5d12f1093420e96661d1c478add9b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46817 } }
I20260619 16:43:53.538645 6562 leader_election.cc:302] T 00000000000000000000000000000000 P 5ba5d12f1093420e96661d1c478add9b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5ba5d12f1093420e96661d1c478add9b; no voters:
I20260619 16:43:53.539222 6562 leader_election.cc:288] T 00000000000000000000000000000000 P 5ba5d12f1093420e96661d1c478add9b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 16:43:53.539321 6567 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5ba5d12f1093420e96661d1c478add9b [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:43:53.539908 6567 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5ba5d12f1093420e96661d1c478add9b [term 1 LEADER]: Becoming Leader. State: Replica: 5ba5d12f1093420e96661d1c478add9b, State: Running, Role: LEADER
I20260619 16:43:53.540423 6567 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5ba5d12f1093420e96661d1c478add9b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ba5d12f1093420e96661d1c478add9b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46817 } }
I20260619 16:43:53.541033 6562 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5ba5d12f1093420e96661d1c478add9b [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:43:53.542886 6568 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5ba5d12f1093420e96661d1c478add9b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5ba5d12f1093420e96661d1c478add9b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ba5d12f1093420e96661d1c478add9b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46817 } } }
I20260619 16:43:53.543395 6568 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5ba5d12f1093420e96661d1c478add9b [sys.catalog]: This master's current role is: LEADER
I20260619 16:43:53.543007 6569 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5ba5d12f1093420e96661d1c478add9b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5ba5d12f1093420e96661d1c478add9b. Latest consensus state: current_term: 1 leader_uuid: "5ba5d12f1093420e96661d1c478add9b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ba5d12f1093420e96661d1c478add9b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46817 } } }
I20260619 16:43:53.544263 6574 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:43:53.544126 6569 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5ba5d12f1093420e96661d1c478add9b [sys.catalog]: This master's current role is: LEADER
I20260619 16:43:53.549499 6574 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:43:53.559319 6574 catalog_manager.cc:1347] Generated new cluster ID: dc389b63eb214c9bb142cff4aef89cd5
I20260619 16:43:53.559451 6574 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:43:53.587953 6574 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 16:43:53.589457 6574 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:43:53.597031 6574 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5ba5d12f1093420e96661d1c478add9b: Generated new TSK 0
I20260619 16:43:53.597937 6574 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887385903558-388-0/krb5kdc/oryx_127.0.97.1.keytab.
Entry for principal oryx/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887385903558-388-0/krb5kdc/oryx_127.0.97.1.keytab.
I20260619 16:43:53.638804 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu
/tmp/dist-test-taskEs2EH8/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887385903558-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887385903558-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887385903558-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887385903558-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46817
--keytab_file=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887385903558-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44537
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887385903558-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887385903558-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887385903558-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:43:53.945142 6588 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:43:53.945411 6588 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:43:53.945529 6588 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:43:53.952772 6588 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:43:53.952883 6588 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:43:53.953027 6588 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260619 16:43:53.962652 6588 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44537
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887385903558-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887385903558-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887385903558-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887385903558-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46817
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887385903558-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:43:53.963920 6588 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:43:53.965180 6588 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:43:53.977502 6593 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:43:53.977490 6594 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:43:53.977669 6596 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:43:53.978122 6588 server_base.cc:1034] running on GCE node
I20260619 16:43:53.978809 6588 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:43:53.979691 6588 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:43:53.980895 6588 hybrid_clock.cc:648] HybridClock initialized: now 1781887433980851 us; error 40 us; skew 500 ppm
I20260619 16:43:53.981200 6588 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:43:53 dist-test-slave-wvz7 krb5kdc[6482](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887433, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:43:53.987129 6588 init.cc:377] Logged in from keytab as oryx/127.0.97.1@KRBTEST.COM (short username oryx)
I20260619 16:43:53.988579 6588 webserver.cc:460] Webserver started at http://127.0.97.1:45013/ using document root <none> and password file <none>
I20260619 16:43:53.989234 6588 fs_manager.cc:359] Metadata directory not provided
I20260619 16:43:53.989351 6588 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:43:53.989626 6588 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:43:53.992440 6588 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887385903558-388-0/minicluster-data/ts-0/data/instance:
uuid: "4e87af70e6654dff8d6f88b6b739ee42"
format_stamp: "Formatted at 2026-06-19 16:43:53 on dist-test-slave-wvz7"
server_key: "f4ca3b4fc3c1cbfc574a36fabd0073fb"
server_key_iv: "bcbd86ec675b13518f137947b870222c"
server_key_version: "encryptionkey@0"
I20260619 16:43:53.993196 6588 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887385903558-388-0/minicluster-data/ts-0/wal/instance:
uuid: "4e87af70e6654dff8d6f88b6b739ee42"
format_stamp: "Formatted at 2026-06-19 16:43:53 on dist-test-slave-wvz7"
server_key: "f4ca3b4fc3c1cbfc574a36fabd0073fb"
server_key_iv: "bcbd86ec675b13518f137947b870222c"
server_key_version: "encryptionkey@0"
I20260619 16:43:53.998226 6588 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260619 16:43:54.001469 6603 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:43:54.002818 6588 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 16:43:54.002974 6588 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887385903558-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887385903558-388-0/minicluster-data/ts-0/wal
uuid: "4e87af70e6654dff8d6f88b6b739ee42"
format_stamp: "Formatted at 2026-06-19 16:43:53 on dist-test-slave-wvz7"
server_key: "f4ca3b4fc3c1cbfc574a36fabd0073fb"
server_key_iv: "bcbd86ec675b13518f137947b870222c"
server_key_version: "encryptionkey@0"
I20260619 16:43:54.003162 6588 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887385903558-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887385903558-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887385903558-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:43:54.019371 6588 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:43:54.023088 6588 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:43:54.023353 6588 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:43:54.024271 6588 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:43:54.025532 6588 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:43:54.025604 6588 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:43:54.025714 6588 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:43:54.025815 6588 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260619 16:43:54.054764 6588 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:40321
I20260619 16:43:54.054809 6715 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:40321 every 8 connection(s)
I20260619 16:43:54.055961 6588 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887385903558-388-0/minicluster-data/ts-0/data/info.pb
I20260619 16:43:54.056560 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu as pid 6588
I20260619 16:43:54.056792 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887385903558-388-0/minicluster-data/ts-0/wal/instance
I20260619 16:43:54.057487 388 external_mini_cluster.cc:1428] Setting key dee01165e9ebe1d67d601cd0972a59d1
Jun 19 16:43:54 dist-test-slave-wvz7 krb5kdc[6482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887433, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for oryx/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260619 16:43:54.075973 6718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:54.058564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:35153 (local address 127.0.97.62:46817)
0619 16:43:54.059001 (+ 437us) server_negotiation.cc:206] Beginning negotiation
0619 16:43:54.059007 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:43:54.060167 (+ 1160us) server_negotiation.cc:407] Connection header received
0619 16:43:54.062124 (+ 1957us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:43:54.062130 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:43:54.062203 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:43:54.062299 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:43:54.063801 (+ 1502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:54.064664 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:54.066213 (+ 1549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:54.066849 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:54.066987 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:43:54.070121 (+ 3134us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:43:54.070128 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:43:54.070133 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:43:54.070171 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:43:54.073101 (+ 2930us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:43:54.073852 (+ 751us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:43:54.073857 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:43:54.073860 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:43:54.073967 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:43:54.074366 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:43:54.074370 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:43:54.074372 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:43:54.074632 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:43:54.074755 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0619 16:43:54.075400 (+ 645us) server_negotiation.cc:299] Negotiation successful
0619 16:43:54.075625 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":322,"thread_start_us":217,"threads_started":1}
I20260619 16:43:54.076905 6719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:54.059204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46817 (local address 127.0.97.1:35153)
0619 16:43:54.060007 (+ 803us) negotiation.cc:107] Waiting for socket to connect
0619 16:43:54.060052 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0619 16:43:54.061852 (+ 1800us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:43:54.062506 (+ 654us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:43:54.062515 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:43:54.063051 (+ 536us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:43:54.063630 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:43:54.063649 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:43:54.064822 (+ 1173us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:43:54.064826 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:43:54.066072 (+ 1246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:43:54.066083 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:43:54.067031 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:43:54.067036 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:43:54.067237 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:43:54.067860 (+ 623us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:43:54.067881 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:43:54.069901 (+ 2020us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:43:54.073221 (+ 3320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:43:54.073229 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:43:54.073245 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:43:54.073727 (+ 482us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:43:54.074061 (+ 334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:43:54.074067 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:43:54.074071 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:43:54.074250 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:43:54.074743 (+ 493us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:43:54.074750 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:43:54.075063 (+ 313us) client_negotiation.cc:769] Sending connection context
0619 16:43:54.075294 (+ 231us) client_negotiation.cc:240] Negotiation successful
0619 16:43:54.075786 (+ 492us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":522,"thread_start_us":333,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.2@KRBTEST.COM" created.
I20260619 16:43:54.078298 6716 heartbeater.cc:347] Connected to a master server at 127.0.97.62:46817
I20260619 16:43:54.078615 6716 heartbeater.cc:464] Registering TS with master...
I20260619 16:43:54.079408 6716 heartbeater.cc:511] Master 127.0.97.62:46817 requested a full tablet report, sending...
I20260619 16:43:54.081414 6527 ts_manager.cc:194] Registered new tserver with Master: 4e87af70e6654dff8d6f88b6b739ee42 (127.0.97.1:40321)
I20260619 16:43:54.082831 6527 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:35153
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887385903558-388-0/krb5kdc/oryx_127.0.97.2.keytab.
Entry for principal oryx/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887385903558-388-0/krb5kdc/oryx_127.0.97.2.keytab.
I20260619 16:43:54.102625 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu
/tmp/dist-test-taskEs2EH8/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887385903558-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887385903558-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887385903558-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887385903558-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46817
--keytab_file=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887385903558-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44537
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887385903558-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887385903558-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887385903558-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:43:54.415109 6722 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:43:54.415366 6722 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:43:54.415455 6722 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:43:54.422389 6722 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:43:54.422485 6722 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:43:54.422600 6722 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260619 16:43:54.431838 6722 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44537
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887385903558-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887385903558-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887385903558-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887385903558-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46817
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887385903558-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:43:54.433097 6722 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:43:54.434372 6722 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:43:54.447062 6727 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:43:54.447039 6730 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:43:54.447005 6728 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:43:54.447640 6722 server_base.cc:1034] running on GCE node
I20260619 16:43:54.448266 6722 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:43:54.449131 6722 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:43:54.450325 6722 hybrid_clock.cc:648] HybridClock initialized: now 1781887434450269 us; error 43 us; skew 500 ppm
I20260619 16:43:54.450613 6722 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:43:54 dist-test-slave-wvz7 krb5kdc[6482](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887434, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:43:54.453469 6722 init.cc:377] Logged in from keytab as oryx/127.0.97.2@KRBTEST.COM (short username oryx)
I20260619 16:43:54.454829 6722 webserver.cc:460] Webserver started at http://127.0.97.2:37687/ using document root <none> and password file <none>
I20260619 16:43:54.455467 6722 fs_manager.cc:359] Metadata directory not provided
I20260619 16:43:54.455578 6722 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:43:54.455852 6722 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:43:54.458595 6722 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887385903558-388-0/minicluster-data/ts-1/data/instance:
uuid: "449f7e56e3754b22a114d8f19ed82f76"
format_stamp: "Formatted at 2026-06-19 16:43:54 on dist-test-slave-wvz7"
server_key: "957a080faef101cc972bc289c8bc2c1e"
server_key_iv: "2e02218e7d0bf01ea06e5f9a5461ee19"
server_key_version: "encryptionkey@0"
I20260619 16:43:54.459302 6722 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887385903558-388-0/minicluster-data/ts-1/wal/instance:
uuid: "449f7e56e3754b22a114d8f19ed82f76"
format_stamp: "Formatted at 2026-06-19 16:43:54 on dist-test-slave-wvz7"
server_key: "957a080faef101cc972bc289c8bc2c1e"
server_key_iv: "2e02218e7d0bf01ea06e5f9a5461ee19"
server_key_version: "encryptionkey@0"
I20260619 16:43:54.464046 6722 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 16:43:54.467218 6737 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:43:54.468453 6722 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 16:43:54.468590 6722 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887385903558-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887385903558-388-0/minicluster-data/ts-1/wal
uuid: "449f7e56e3754b22a114d8f19ed82f76"
format_stamp: "Formatted at 2026-06-19 16:43:54 on dist-test-slave-wvz7"
server_key: "957a080faef101cc972bc289c8bc2c1e"
server_key_iv: "2e02218e7d0bf01ea06e5f9a5461ee19"
server_key_version: "encryptionkey@0"
I20260619 16:43:54.468787 6722 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887385903558-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887385903558-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887385903558-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:43:54.486835 6722 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:43:54.490742 6722 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:43:54.491015 6722 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:43:54.491940 6722 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:43:54.493240 6722 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:43:54.493314 6722 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:43:54.493428 6722 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:43:54.493479 6722 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:43:54.523068 6722 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:34331
I20260619 16:43:54.523110 6849 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:34331 every 8 connection(s)
I20260619 16:43:54.524201 6722 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887385903558-388-0/minicluster-data/ts-1/data/info.pb
I20260619 16:43:54.533914 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu as pid 6722
I20260619 16:43:54.534057 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887385903558-388-0/minicluster-data/ts-1/wal/instance
I20260619 16:43:54.534530 388 external_mini_cluster.cc:1428] Setting key bf50222584db2be6bd01e8a3e2960634
Jun 19 16:43:54 dist-test-slave-wvz7 krb5kdc[6482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887434, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260619 16:43:54.542693 6718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:54.526454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:50235 (local address 127.0.97.62:46817)
0619 16:43:54.526658 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0619 16:43:54.526664 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:43:54.527805 (+ 1141us) server_negotiation.cc:407] Connection header received
0619 16:43:54.529454 (+ 1649us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:43:54.529458 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:43:54.529523 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:43:54.529609 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:43:54.531097 (+ 1488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:54.531779 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:54.533189 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:54.533564 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:54.533670 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:43:54.537195 (+ 3525us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:43:54.537209 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:43:54.537214 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:43:54.537249 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:43:54.540002 (+ 2753us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:43:54.540633 (+ 631us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:43:54.540638 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:43:54.540641 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:43:54.540779 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:43:54.541125 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:43:54.541129 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:43:54.541132 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:43:54.541429 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:43:54.541547 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0619 16:43:54.542219 (+ 672us) server_negotiation.cc:299] Negotiation successful
0619 16:43:54.542414 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260619 16:43:54.543797 6852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:54.526973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46817 (local address 127.0.97.2:50235)
0619 16:43:54.527649 (+ 676us) negotiation.cc:107] Waiting for socket to connect
0619 16:43:54.527685 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0619 16:43:54.529217 (+ 1532us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:43:54.529894 (+ 677us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:43:54.529902 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:43:54.530400 (+ 498us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:43:54.530959 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:43:54.530976 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:43:54.531893 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:43:54.531897 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:43:54.533083 (+ 1186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:43:54.533095 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:43:54.533807 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:43:54.533816 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:43:54.534024 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:43:54.534786 (+ 762us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:43:54.534811 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:43:54.537001 (+ 2190us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:43:54.540127 (+ 3126us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:43:54.540133 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:43:54.540144 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:43:54.540521 (+ 377us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:43:54.540876 (+ 355us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:43:54.540879 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:43:54.540881 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:43:54.541040 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:43:54.541541 (+ 501us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:43:54.541546 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:43:54.541884 (+ 338us) client_negotiation.cc:769] Sending connection context
0619 16:43:54.542116 (+ 232us) client_negotiation.cc:240] Negotiation successful
0619 16:43:54.542482 (+ 366us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":453,"thread_start_us":273,"threads_started":1}
I20260619 16:43:54.545066 6850 heartbeater.cc:347] Connected to a master server at 127.0.97.62:46817
I20260619 16:43:54.545346 6850 heartbeater.cc:464] Registering TS with master...
I20260619 16:43:54.546065 6850 heartbeater.cc:511] Master 127.0.97.62:46817 requested a full tablet report, sending...
I20260619 16:43:54.547389 6527 ts_manager.cc:194] Registered new tserver with Master: 449f7e56e3754b22a114d8f19ed82f76 (127.0.97.2:34331)
I20260619 16:43:54.548372 6527 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:50235
WARNING: no policy specified for oryx/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887385903558-388-0/krb5kdc/oryx_127.0.97.3.keytab.
Entry for principal oryx/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887385903558-388-0/krb5kdc/oryx_127.0.97.3.keytab.
I20260619 16:43:54.574692 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu
/tmp/dist-test-taskEs2EH8/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887385903558-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887385903558-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887385903558-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887385903558-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46817
--keytab_file=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887385903558-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44537
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887385903558-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887385903558-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887385903558-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:43:54.874460 6855 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:43:54.874733 6855 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:43:54.874851 6855 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:43:54.881896 6855 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:43:54.882004 6855 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:43:54.882149 6855 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260619 16:43:54.891535 6855 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44537
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887385903558-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887385903558-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887385903558-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887385903558-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46817
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887385903558-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:43:54.892737 6855 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:43:54.894040 6855 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:43:54.905997 6860 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:43:54.906062 6861 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:43:54.906144 6863 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:43:54.907419 6855 server_base.cc:1034] running on GCE node
I20260619 16:43:54.907994 6855 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:43:54.908814 6855 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:43:54.909996 6855 hybrid_clock.cc:648] HybridClock initialized: now 1781887434909944 us; error 47 us; skew 500 ppm
I20260619 16:43:54.910279 6855 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:43:54 dist-test-slave-wvz7 krb5kdc[6482](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887434, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:43:54.913085 6855 init.cc:377] Logged in from keytab as oryx/127.0.97.3@KRBTEST.COM (short username oryx)
I20260619 16:43:54.914460 6855 webserver.cc:460] Webserver started at http://127.0.97.3:37901/ using document root <none> and password file <none>
I20260619 16:43:54.915091 6855 fs_manager.cc:359] Metadata directory not provided
I20260619 16:43:54.915197 6855 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:43:54.915452 6855 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:43:54.918130 6855 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887385903558-388-0/minicluster-data/ts-2/data/instance:
uuid: "399a36acb0b447e8b9c0d6992bc86ea2"
format_stamp: "Formatted at 2026-06-19 16:43:54 on dist-test-slave-wvz7"
server_key: "afbfae96eaddf174673547acc725f829"
server_key_iv: "7d5232cc81be8d93525f98fdc48b6ee9"
server_key_version: "encryptionkey@0"
I20260619 16:43:54.918831 6855 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887385903558-388-0/minicluster-data/ts-2/wal/instance:
uuid: "399a36acb0b447e8b9c0d6992bc86ea2"
format_stamp: "Formatted at 2026-06-19 16:43:54 on dist-test-slave-wvz7"
server_key: "afbfae96eaddf174673547acc725f829"
server_key_iv: "7d5232cc81be8d93525f98fdc48b6ee9"
server_key_version: "encryptionkey@0"
I20260619 16:43:54.923558 6855 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 16:43:54.926867 6870 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:43:54.928095 6855 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260619 16:43:54.928233 6855 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887385903558-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887385903558-388-0/minicluster-data/ts-2/wal
uuid: "399a36acb0b447e8b9c0d6992bc86ea2"
format_stamp: "Formatted at 2026-06-19 16:43:54 on dist-test-slave-wvz7"
server_key: "afbfae96eaddf174673547acc725f829"
server_key_iv: "7d5232cc81be8d93525f98fdc48b6ee9"
server_key_version: "encryptionkey@0"
I20260619 16:43:54.928399 6855 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887385903558-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887385903558-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887385903558-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:43:54.943315 6855 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:43:54.947016 6855 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:43:54.947255 6855 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:43:54.948123 6855 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:43:54.949306 6855 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:43:54.949375 6855 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:43:54.949477 6855 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:43:54.949529 6855 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:43:54.977789 6855 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:40501
I20260619 16:43:54.977857 6982 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:40501 every 8 connection(s)
I20260619 16:43:54.978981 6855 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887385903558-388-0/minicluster-data/ts-2/data/info.pb
I20260619 16:43:54.980883 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu as pid 6855
I20260619 16:43:54.981048 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887385903558-388-0/minicluster-data/ts-2/wal/instance
I20260619 16:43:54.981587 388 external_mini_cluster.cc:1428] Setting key 859584bcc0f7db5e4d1f6d86ed0fd203
Jun 19 16:43:54 dist-test-slave-wvz7 krb5kdc[6482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887434, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260619 16:43:54.997682 6718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:54.981396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:33179 (local address 127.0.97.62:46817)
0619 16:43:54.981552 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0619 16:43:54.981556 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:43:54.983006 (+ 1450us) server_negotiation.cc:407] Connection header received
0619 16:43:54.985114 (+ 2108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:43:54.985119 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:43:54.985195 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:43:54.985287 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:43:54.986712 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:54.987499 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:54.988969 (+ 1470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:54.989374 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:54.989480 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:43:54.992719 (+ 3239us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:43:54.992726 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:43:54.992730 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:43:54.992762 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:43:54.994956 (+ 2194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:43:54.995676 (+ 720us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:43:54.995682 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:43:54.995685 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:43:54.995795 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:43:54.996220 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:43:54.996241 (+ 21us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:43:54.996244 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:43:54.996535 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:43:54.996645 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0619 16:43:54.997286 (+ 641us) server_negotiation.cc:299] Negotiation successful
0619 16:43:54.997444 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260619 16:43:54.998593 6985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:54.982091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46817 (local address 127.0.97.3:33179)
0619 16:43:54.982860 (+ 769us) negotiation.cc:107] Waiting for socket to connect
0619 16:43:54.982918 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0619 16:43:54.984880 (+ 1962us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:43:54.985513 (+ 633us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:43:54.985522 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:43:54.986033 (+ 511us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:43:54.986562 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:43:54.986577 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:43:54.987648 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:43:54.987652 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:43:54.988847 (+ 1195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:43:54.988857 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:43:54.989565 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:43:54.989571 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:43:54.989862 (+ 291us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:43:54.990460 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:43:54.990481 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:43:54.992527 (+ 2046us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:43:54.995071 (+ 2544us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:43:54.995080 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:43:54.995096 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:43:54.995568 (+ 472us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:43:54.995889 (+ 321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:43:54.995894 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:43:54.995898 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:43:54.996113 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:43:54.996638 (+ 525us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:43:54.996643 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:43:54.996937 (+ 294us) client_negotiation.cc:769] Sending connection context
0619 16:43:54.997144 (+ 207us) client_negotiation.cc:240] Negotiation successful
0619 16:43:54.997498 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":444,"thread_start_us":269,"threads_started":1}
I20260619 16:43:54.999840 6983 heartbeater.cc:347] Connected to a master server at 127.0.97.62:46817
I20260619 16:43:55.000084 6983 heartbeater.cc:464] Registering TS with master...
I20260619 16:43:55.000727 6983 heartbeater.cc:511] Master 127.0.97.62:46817 requested a full tablet report, sending...
I20260619 16:43:55.002022 6527 ts_manager.cc:194] Registered new tserver with Master: 399a36acb0b447e8b9c0d6992bc86ea2 (127.0.97.3:40501)
I20260619 16:43:55.002774 6527 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:33179
I20260619 16:43:55.008565 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.97.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887385903558-388-0/krb5kdc/oryx_127.0.97.61.keytab.
Entry for principal oryx/127.0.97.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887385903558-388-0/krb5kdc/oryx_127.0.97.61.keytab.
I20260619 16:43:55.048784 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu
/tmp/dist-test-taskEs2EH8/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887385903558-388-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887385903558-388-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887385903558-388-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887385903558-388-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:42337
--webserver_interface=127.0.97.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887385903558-388-0/krb5kdc/oryx_127.0.97.61.keytab
--principal=oryx/127.0.97.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44537
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:46817,127.0.97.61:42337
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887385903558-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887385903558-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887385903558-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 16:43:55.085717 6716 heartbeater.cc:503] Master 127.0.97.62:46817 was elected leader, sending a full tablet report...
W20260619 16:43:55.352939 6988 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:43:55.353215 6988 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:43:55.353340 6988 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:43:55.360432 6988 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:43:55.360525 6988 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:43:55.360599 6988 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:43:55.360667 6988 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:43:55.360728 6988 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:43:55.370354 6988 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44537
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887385903558-388-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887385903558-388-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:46817,127.0.97.61:42337
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887385903558-388-0/krb5kdc/oryx_127.0.97.61.keytab
--principal=oryx/127.0.97.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:42337
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887385903558-388-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.97.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887385903558-388-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
W20260619 16:43:55.370682 6988 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.0.97.62:46817,127.0.97.61:42337'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260619 16:43:55.371778 6988 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:43:55.373075 6988 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:43:55.383718 6993 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:43:55.383726 6994 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:43:55.383767 6996 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:43:55.384800 6988 server_base.cc:1034] running on GCE node
I20260619 16:43:55.385469 6988 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:43:55.386744 6988 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:43:55.387948 6988 hybrid_clock.cc:648] HybridClock initialized: now 1781887435387885 us; error 43 us; skew 500 ppm
I20260619 16:43:55.388242 6988 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:43:55 dist-test-slave-wvz7 krb5kdc[6482](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887435, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:43:55.391019 6988 init.cc:377] Logged in from keytab as oryx/127.0.97.61@KRBTEST.COM (short username oryx)
I20260619 16:43:55.392345 6988 webserver.cc:460] Webserver started at http://127.0.97.61:35663/ using document root <none> and password file <none>
I20260619 16:43:55.392956 6988 fs_manager.cc:359] Metadata directory not provided
I20260619 16:43:55.393047 6988 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:43:55.393321 6988 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:43:55.396018 6988 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887385903558-388-0/minicluster-data/master-1/data/instance:
uuid: "85d1444e474f4e189618fb22f5a42908"
format_stamp: "Formatted at 2026-06-19 16:43:55 on dist-test-slave-wvz7"
server_key: "c67ccda032cbde5cc15fec7ba65504f1"
server_key_iv: "265dcd58e69572ada7ceccf68de61e88"
server_key_version: "encryptionkey@0"
I20260619 16:43:55.396723 6988 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887385903558-388-0/minicluster-data/master-1/wal/instance:
uuid: "85d1444e474f4e189618fb22f5a42908"
format_stamp: "Formatted at 2026-06-19 16:43:55 on dist-test-slave-wvz7"
server_key: "c67ccda032cbde5cc15fec7ba65504f1"
server_key_iv: "265dcd58e69572ada7ceccf68de61e88"
server_key_version: "encryptionkey@0"
I20260619 16:43:55.401363 6988 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.003s
I20260619 16:43:55.405591 7003 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:43:55.407004 6988 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 16:43:55.407158 6988 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887385903558-388-0/minicluster-data/master-1/data,/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887385903558-388-0/minicluster-data/master-1/wal
uuid: "85d1444e474f4e189618fb22f5a42908"
format_stamp: "Formatted at 2026-06-19 16:43:55 on dist-test-slave-wvz7"
server_key: "c67ccda032cbde5cc15fec7ba65504f1"
server_key_iv: "265dcd58e69572ada7ceccf68de61e88"
server_key_version: "encryptionkey@0"
I20260619 16:43:55.407342 6988 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887385903558-388-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887385903558-388-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887385903558-388-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:43:55.427935 6988 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:43:55.431530 6988 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:43:55.431748 6988 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:43:55.448244 6988 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.61:42337
I20260619 16:43:55.448225 7054 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.61:42337 every 8 connection(s)
I20260619 16:43:55.449445 6988 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887385903558-388-0/minicluster-data/master-1/data/info.pb
I20260619 16:43:55.453011 7055 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:43:55.455547 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu as pid 6988
I20260619 16:43:55.460807 7055 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46817 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 16:43:55 dist-test-slave-wvz7 krb5kdc[6482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887433, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
Jun 19 16:43:55 dist-test-slave-wvz7 krb5kdc[6482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887435, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.61@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260619 16:43:55.478487 6718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:55.463759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46046 (local address 127.0.97.62:46817)
0619 16:43:55.463903 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0619 16:43:55.463907 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:43:55.464627 (+ 720us) server_negotiation.cc:407] Connection header received
0619 16:43:55.464867 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:43:55.464871 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:43:55.464931 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:43:55.465000 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:43:55.466419 (+ 1419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:55.467429 (+ 1010us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:55.469308 (+ 1879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:55.469678 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:55.469840 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:43:55.473232 (+ 3392us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:43:55.473237 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:43:55.473241 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:43:55.473270 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:43:55.475829 (+ 2559us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:43:55.476563 (+ 734us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:43:55.476568 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:43:55.476571 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:43:55.476696 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:43:55.477027 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:43:55.477032 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:43:55.477036 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:43:55.477331 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:43:55.477456 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0619 16:43:55.477980 (+ 524us) server_negotiation.cc:299] Negotiation successful
0619 16:43:55.478189 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260619 16:43:55.479425 7058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:55.457882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42084 (local address 127.0.97.61:42337)
0619 16:43:55.458600 (+ 718us) server_negotiation.cc:206] Beginning negotiation
0619 16:43:55.458611 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 16:43:55.458657 (+ 46us) server_negotiation.cc:407] Connection header received
0619 16:43:55.460197 (+ 1540us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:43:55.460219 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:43:55.460591 (+ 372us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:43:55.460926 (+ 335us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:43:55.462818 (+ 1892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:55.464593 (+ 1775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:55.466369 (+ 1776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:55.467331 (+ 962us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:55.467503 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:43:55.472310 (+ 4807us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:43:55.472322 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:43:55.472337 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:43:55.472380 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:43:55.475392 (+ 3012us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:43:55.476055 (+ 663us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:43:55.476062 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:43:55.476064 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:43:55.476249 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:43:55.476650 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:43:55.476653 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:43:55.476655 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:43:55.477145 (+ 490us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:43:55.477437 (+ 292us) server_negotiation.cc:1035] Waiting for connection context
0619 16:43:55.477870 (+ 433us) server_negotiation.cc:299] Negotiation successful
0619 16:43:55.478272 (+ 402us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":521,"thread_start_us":308,"threads_started":1}
I20260619 16:43:55.479440 7059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:55.463801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46817 (local address 127.0.0.1:46046)
0619 16:43:55.464494 (+ 693us) negotiation.cc:107] Waiting for socket to connect
0619 16:43:55.464528 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0619 16:43:55.464722 (+ 194us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:43:55.465081 (+ 359us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:43:55.465088 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:43:55.465922 (+ 834us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:43:55.466253 (+ 331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:43:55.466265 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:43:55.467571 (+ 1306us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:43:55.467576 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:43:55.469165 (+ 1589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:43:55.469182 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:43:55.469931 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:43:55.469940 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:43:55.470131 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:43:55.470693 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:43:55.470720 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:43:55.473017 (+ 2297us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:43:55.476024 (+ 3007us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:43:55.476034 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:43:55.476046 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:43:55.476458 (+ 412us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:43:55.476795 (+ 337us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:43:55.476799 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:43:55.476803 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:43:55.476933 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:43:55.477447 (+ 514us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:43:55.477454 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:43:55.477638 (+ 184us) client_negotiation.cc:769] Sending connection context
0619 16:43:55.477885 (+ 247us) client_negotiation.cc:240] Negotiation successful
0619 16:43:55.478429 (+ 544us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":504,"mutex_wait_us":120,"thread_start_us":377,"threads_started":1}
I20260619 16:43:55.481241 7055 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42337 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 16:43:55 dist-test-slave-wvz7 krb5kdc[6482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887435, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.61@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
I20260619 16:43:55.492326 7059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:55.482043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:42337 (local address 127.0.0.1:42092)
0619 16:43:55.482199 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0619 16:43:55.482216 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 16:43:55.482298 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:43:55.482580 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:43:55.482584 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:43:55.482872 (+ 288us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:43:55.482992 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:43:55.482999 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:43:55.483876 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:43:55.483880 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:43:55.484710 (+ 830us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:43:55.484719 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:43:55.485326 (+ 607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:43:55.485330 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:43:55.485465 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:43:55.485979 (+ 514us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:43:55.485996 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:43:55.487849 (+ 1853us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:43:55.490314 (+ 2465us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:43:55.490317 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:43:55.490319 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:43:55.490656 (+ 337us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:43:55.490986 (+ 330us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:43:55.490989 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:43:55.490992 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:43:55.491143 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:43:55.491637 (+ 494us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:43:55.491640 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:43:55.491761 (+ 121us) client_negotiation.cc:769] Sending connection context
0619 16:43:55.491878 (+ 117us) client_negotiation.cc:240] Negotiation successful
0619 16:43:55.492079 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56}
I20260619 16:43:55.492585 7058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:55.482022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42092 (local address 127.0.97.61:42337)
0619 16:43:55.482177 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0619 16:43:55.482183 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:43:55.482271 (+ 88us) server_negotiation.cc:407] Connection header received
0619 16:43:55.482389 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:43:55.482393 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:43:55.482457 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:43:55.482541 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:43:55.483096 (+ 555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:55.483774 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:55.484841 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:55.485207 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:55.485305 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:43:55.488003 (+ 2698us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:43:55.488007 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:43:55.488010 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:43:55.488035 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:43:55.490200 (+ 2165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:43:55.490776 (+ 576us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:43:55.490780 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:43:55.490782 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:43:55.490876 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:43:55.491269 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:43:55.491273 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:43:55.491275 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:43:55.491536 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:43:55.491668 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0619 16:43:55.491936 (+ 268us) server_negotiation.cc:299] Negotiation successful
0619 16:43:55.492257 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":117,"server-negotiator.queue_time_us":44}
I20260619 16:43:55.496790 7055 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 85d1444e474f4e189618fb22f5a42908: Bootstrap starting.
I20260619 16:43:55.499752 7055 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 85d1444e474f4e189618fb22f5a42908: Neither blocks nor log segments found. Creating new log.
I20260619 16:43:55.500636 7055 log.cc:824] T 00000000000000000000000000000000 P 85d1444e474f4e189618fb22f5a42908: Log is configured to *not* fsync() on all Append() calls
I20260619 16:43:55.503379 7055 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 85d1444e474f4e189618fb22f5a42908: No bootstrap required, opened a new log
I20260619 16:43:55.513639 7055 raft_consensus.cc:348] T 00000000000000000000000000000000 P 85d1444e474f4e189618fb22f5a42908 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ba5d12f1093420e96661d1c478add9b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46817 } } peers { permanent_uuid: "85d1444e474f4e189618fb22f5a42908" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42337 } }
I20260619 16:43:55.514065 7055 raft_consensus.cc:374] T 00000000000000000000000000000000 P 85d1444e474f4e189618fb22f5a42908 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:43:55.514155 7055 raft_consensus.cc:729] T 00000000000000000000000000000000 P 85d1444e474f4e189618fb22f5a42908 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85d1444e474f4e189618fb22f5a42908, State: Initialized, Role: FOLLOWER
I20260619 16:43:55.514767 7055 consensus_queue.cc:260] T 00000000000000000000000000000000 P 85d1444e474f4e189618fb22f5a42908 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ba5d12f1093420e96661d1c478add9b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46817 } } peers { permanent_uuid: "85d1444e474f4e189618fb22f5a42908" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42337 } }
I20260619 16:43:55.515539 7061 sys_catalog.cc:455] T 00000000000000000000000000000000 P 85d1444e474f4e189618fb22f5a42908 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ba5d12f1093420e96661d1c478add9b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46817 } } peers { permanent_uuid: "85d1444e474f4e189618fb22f5a42908" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42337 } } }
I20260619 16:43:55.515823 7061 sys_catalog.cc:458] T 00000000000000000000000000000000 P 85d1444e474f4e189618fb22f5a42908 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 16:43:55.516463 7055 sys_catalog.cc:564] T 00000000000000000000000000000000 P 85d1444e474f4e189618fb22f5a42908 [sys.catalog]: configured and running, proceeding with master startup.
W20260619 16:43:55.524780 7072 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 85d1444e474f4e189618fb22f5a42908: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 16:43:55.524981 7072 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 16:43:55.527045 6988 master_runner.cc:428] Detected that this master 85d1444e474f4e189618fb22f5a42908 is joining an existing cluster
I20260619 16:43:55.527139 6988 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.97.61:42337
I20260619 16:43:55.546752 6718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:55.531358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46054 (local address 127.0.97.62:46817)
0619 16:43:55.531493 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0619 16:43:55.531497 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:43:55.533259 (+ 1762us) server_negotiation.cc:407] Connection header received
0619 16:43:55.533874 (+ 615us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:43:55.533879 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:43:55.533952 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:43:55.534017 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:43:55.535355 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:55.536373 (+ 1018us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:55.538338 (+ 1965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:55.538861 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:55.538951 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:43:55.541218 (+ 2267us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:43:55.541224 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:43:55.541229 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:43:55.541269 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:43:55.544289 (+ 3020us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:43:55.544858 (+ 569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:43:55.544864 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:43:55.544868 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:43:55.545010 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:43:55.545339 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:43:55.545345 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:43:55.545349 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:43:55.545621 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:43:55.545805 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0619 16:43:55.546201 (+ 396us) server_negotiation.cc:299] Negotiation successful
0619 16:43:55.546409 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260619 16:43:55.546866 7058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:55.531064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42104 (local address 127.0.97.61:42337)
0619 16:43:55.531340 (+ 276us) server_negotiation.cc:206] Beginning negotiation
0619 16:43:55.531346 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:43:55.532358 (+ 1012us) server_negotiation.cc:407] Connection header received
0619 16:43:55.532489 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:43:55.532494 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:43:55.532568 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:43:55.532659 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:43:55.534175 (+ 1516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:55.535345 (+ 1170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:55.537204 (+ 1859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:55.537992 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:55.538085 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:43:55.540447 (+ 2362us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:43:55.540453 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:43:55.540457 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:43:55.540488 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:43:55.543987 (+ 3499us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:43:55.544763 (+ 776us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:43:55.544769 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:43:55.544772 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:43:55.544916 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:43:55.545294 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:43:55.545299 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:43:55.545302 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:43:55.545641 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:43:55.545816 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0619 16:43:55.546243 (+ 427us) server_negotiation.cc:299] Negotiation successful
0619 16:43:55.546488 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":48,"server-negotiator.queue_time_us":63}
I20260619 16:43:55.547236 7079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:55.531805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46817 (local address 127.0.0.1:46054)
0619 16:43:55.533165 (+ 1360us) negotiation.cc:107] Waiting for socket to connect
0619 16:43:55.533188 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0619 16:43:55.533701 (+ 513us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:43:55.534210 (+ 509us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:43:55.534214 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:43:55.534891 (+ 677us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:43:55.535205 (+ 314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:43:55.535216 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:43:55.536569 (+ 1353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:43:55.536574 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:43:55.538188 (+ 1614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:43:55.538200 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:43:55.539206 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:43:55.539211 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:43:55.539382 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:43:55.539905 (+ 523us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:43:55.539924 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:43:55.541037 (+ 1113us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:43:55.544408 (+ 3371us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:43:55.544413 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:43:55.544417 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:43:55.544763 (+ 346us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:43:55.545120 (+ 357us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:43:55.545124 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:43:55.545127 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:43:55.545258 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:43:55.545725 (+ 467us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:43:55.545729 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:43:55.545953 (+ 224us) client_negotiation.cc:769] Sending connection context
0619 16:43:55.546081 (+ 128us) client_negotiation.cc:240] Negotiation successful
0619 16:43:55.546693 (+ 612us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1229,"mutex_wait_us":190,"thread_start_us":773,"threads_started":1}
I20260619 16:43:55.547479 7078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:55.531037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:42337 (local address 127.0.0.1:42104)
0619 16:43:55.532187 (+ 1150us) negotiation.cc:107] Waiting for socket to connect
0619 16:43:55.532213 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0619 16:43:55.532364 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:43:55.532819 (+ 455us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:43:55.532827 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:43:55.533498 (+ 671us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:43:55.534040 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:43:55.534054 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:43:55.535547 (+ 1493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:43:55.535552 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:43:55.537066 (+ 1514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:43:55.537080 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:43:55.538362 (+ 1282us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:43:55.538368 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:43:55.538534 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:43:55.539112 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:43:55.539134 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:43:55.540269 (+ 1135us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:43:55.544118 (+ 3849us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:43:55.544123 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:43:55.544126 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:43:55.544658 (+ 532us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:43:55.545008 (+ 350us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:43:55.545013 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:43:55.545017 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:43:55.545171 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:43:55.545908 (+ 737us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:43:55.545912 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:43:55.546037 (+ 125us) client_negotiation.cc:769] Sending connection context
0619 16:43:55.546341 (+ 304us) client_negotiation.cc:240] Negotiation successful
0619 16:43:55.546633 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":993,"mutex_wait_us":55,"thread_start_us":846,"threads_started":1}
I20260619 16:43:55.550472 7061 raft_consensus.cc:482] T 00000000000000000000000000000000 P 85d1444e474f4e189618fb22f5a42908 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 16:43:55.550697 6850 heartbeater.cc:503] Master 127.0.97.62:46817 was elected leader, sending a full tablet report...
I20260619 16:43:55.550722 7061 raft_consensus.cc:504] T 00000000000000000000000000000000 P 85d1444e474f4e189618fb22f5a42908 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ba5d12f1093420e96661d1c478add9b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46817 } } peers { permanent_uuid: "85d1444e474f4e189618fb22f5a42908" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42337 } }
I20260619 16:43:55.552227 7061 leader_election.cc:288] T 00000000000000000000000000000000 P 85d1444e474f4e189618fb22f5a42908 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5ba5d12f1093420e96661d1c478add9b (127.0.97.62:46817)
I20260619 16:43:55.553040 6537 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "85d1444e474f4e189618fb22f5a42908" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ba5d12f1093420e96661d1c478add9b" is_pre_election: true
I20260619 16:43:55.554240 7007 leader_election.cc:302] T 00000000000000000000000000000000 P 85d1444e474f4e189618fb22f5a42908 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: 85d1444e474f4e189618fb22f5a42908; no voters: 5ba5d12f1093420e96661d1c478add9b
I20260619 16:43:55.555004 7061 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 85d1444e474f4e189618fb22f5a42908 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
WARNING: no policy specified for oryx/127.0.97.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.60@KRBTEST.COM" created.
Jun 19 16:43:55 dist-test-slave-wvz7 krb5kdc[6482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887433, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.62@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
I20260619 16:43:55.576223 7081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:55.554842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:42337 (local address 127.0.0.1:42116)
0619 16:43:55.558010 (+ 3168us) negotiation.cc:107] Waiting for socket to connect
0619 16:43:55.558043 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0619 16:43:55.558333 (+ 290us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:43:55.558761 (+ 428us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:43:55.558768 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:43:55.559432 (+ 664us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:43:55.559649 (+ 217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:43:55.559658 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:43:55.562933 (+ 3275us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:43:55.562939 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:43:55.564459 (+ 1520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:43:55.564474 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:43:55.565691 (+ 1217us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:43:55.565698 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:43:55.565953 (+ 255us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:43:55.566608 (+ 655us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:43:55.566631 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:43:55.569268 (+ 2637us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:43:55.573163 (+ 3895us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:43:55.573175 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:43:55.573182 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:43:55.573780 (+ 598us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:43:55.574149 (+ 369us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:43:55.574154 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:43:55.574158 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:43:55.574323 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:43:55.575204 (+ 881us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:43:55.575213 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:43:55.575413 (+ 200us) client_negotiation.cc:769] Sending connection context
0619 16:43:55.575626 (+ 213us) client_negotiation.cc:240] Negotiation successful
0619 16:43:55.575879 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2995,"thread_start_us":2889,"threads_started":1}
I20260619 16:43:55.576313 7058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:55.555138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42116 (local address 127.0.97.61:42337)
0619 16:43:55.555425 (+ 287us) server_negotiation.cc:206] Beginning negotiation
0619 16:43:55.555431 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:43:55.558221 (+ 2790us) server_negotiation.cc:407] Connection header received
0619 16:43:55.558509 (+ 288us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:43:55.558514 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:43:55.558613 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:43:55.558688 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:43:55.561851 (+ 3163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:55.562781 (+ 930us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:55.564634 (+ 1853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:55.565351 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:55.565443 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:43:55.569456 (+ 4013us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:43:55.569462 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:43:55.569467 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:43:55.569501 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:43:55.573038 (+ 3537us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:43:55.573882 (+ 844us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:43:55.573888 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:43:55.573892 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:43:55.574040 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:43:55.574456 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:43:55.574461 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:43:55.574465 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:43:55.574772 (+ 307us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:43:55.574921 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0619 16:43:55.575753 (+ 832us) server_negotiation.cc:299] Negotiation successful
0619 16:43:55.575977 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":172}
I20260619 16:43:55.578150 6527 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.97.61:42337: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "85d1444e474f4e189618fb22f5a42908"
member_type: NON_VOTER
last_known_addr {
host: "127.0.97.61"
port: 42337
}
attrs {
promote: true
}
}
dest_uuid: "5ba5d12f1093420e96661d1c478add9b"
cas_config_opid_index: -1
I20260619 16:43:55.579236 6527 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5ba5d12f1093420e96661d1c478add9b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "5ba5d12f1093420e96661d1c478add9b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46817 } } peers { permanent_uuid: "85d1444e474f4e189618fb22f5a42908" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 42337 } attrs { promote: true } }
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887385903558-388-0/krb5kdc/oryx_127.0.97.60.keytab.
Entry for principal oryx/127.0.97.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887385903558-388-0/krb5kdc/oryx_127.0.97.60.keytab.
I20260619 16:43:55.583014 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu
/tmp/dist-test-taskEs2EH8/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887385903558-388-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887385903558-388-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887385903558-388-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887385903558-388-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:35535
--webserver_interface=127.0.97.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887385903558-388-0/krb5kdc/oryx_127.0.97.60.keytab
--principal=oryx/127.0.97.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44537
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:46817,127.0.97.61:42337,127.0.97.60:35535
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887385903558-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887385903558-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887385903558-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 16:43:55.582516 7084 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 5ba5d12f1093420e96661d1c478add9b [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 85d1444e474f4e189618fb22f5a42908 (127.0.97.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "5ba5d12f1093420e96661d1c478add9b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46817 } } peers { permanent_uuid: "85d1444e474f4e189618fb22f5a42908" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 42337 } attrs { promote: true } } }
I20260619 16:43:55.586659 7084 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.97.61:42337
I20260619 16:43:55.587487 6988 master.cc:537] Master@127.0.97.61:42337 shutting down...
I20260619 16:43:55.588615 7084 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5ba5d12f1093420e96661d1c478add9b [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "5ba5d12f1093420e96661d1c478add9b" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "5ba5d12f1093420e96661d1c478add9b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46817 } } peers { permanent_uuid: "85d1444e474f4e189618fb22f5a42908" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 42337 } attrs { promote: true } } }
I20260619 16:43:55.588912 7084 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5ba5d12f1093420e96661d1c478add9b [sys.catalog]: This master's current role is: LEADER
W20260619 16:43:55.589937 6512 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
I20260619 16:43:55.600235 6988 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 85d1444e474f4e189618fb22f5a42908 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260619 16:43:55.601099 6988 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 85d1444e474f4e189618fb22f5a42908 [term 0 FOLLOWER]: Raft consensus is shut down!
W20260619 16:43:55.601125 6512 consensus_peers.cc:487] T 00000000000000000000000000000000 P 5ba5d12f1093420e96661d1c478add9b -> Peer 85d1444e474f4e189618fb22f5a42908 (127.0.97.61:42337): Couldn't send request to peer 85d1444e474f4e189618fb22f5a42908. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260619 16:43:55.601368 6988 tablet_replica.cc:330] T 00000000000000000000000000000000 P 85d1444e474f4e189618fb22f5a42908: stopping tablet replica
I20260619 16:43:55.608484 6988 master.cc:552] Master@127.0.97.61:42337 shutdown complete.
I20260619 16:43:55.608633 6988 master_runner.cc:305] Clearing existing system tablet
I20260619 16:43:55.609110 6988 fs_manager.cc:359] Metadata directory not provided
I20260619 16:43:55.609254 6988 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:43:55.614430 6988 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260619 16:43:55.616143 7090 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:43:55.616509 6988 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887385903558-388-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:43:55.616968 6988 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 16:43:55.617103 6988 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887385903558-388-0/minicluster-data/master-1/data,/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887385903558-388-0/minicluster-data/master-1/wal
uuid: "85d1444e474f4e189618fb22f5a42908"
format_stamp: "Formatted at 2026-06-19 16:43:55 on dist-test-slave-wvz7"
server_key: "c67ccda032cbde5cc15fec7ba65504f1"
server_key_iv: "265dcd58e69572ada7ceccf68de61e88"
server_key_version: "encryptionkey@0"
I20260619 16:43:55.622272 6988 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 85d1444e474f4e189618fb22f5a42908: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:43:55.629101 6988 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 85d1444e474f4e189618fb22f5a42908: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260619 16:43:55.629359 6988 log.cc:1196] T 00000000000000000000000000000000 P 85d1444e474f4e189618fb22f5a42908: Deleting WAL directory at /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887385903558-388-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260619 16:43:55.629848 6988 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 85d1444e474f4e189618fb22f5a42908: Deleting consensus metadata
I20260619 16:43:55.630155 6988 master_runner.cc:315] Copying system tablet from 127.0.97.62:46817
I20260619 16:43:55.632555 6988 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 85d1444e474f4e189618fb22f5a42908: tablet copy: Beginning tablet copy session from remote peer at address 127.0.97.62:46817
I20260619 16:43:55.648037 6718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:55.633406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46064 (local address 127.0.97.62:46817)
0619 16:43:55.633595 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0619 16:43:55.633601 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:43:55.634078 (+ 477us) server_negotiation.cc:407] Connection header received
0619 16:43:55.634363 (+ 285us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:43:55.634370 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:43:55.634465 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:43:55.634536 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:43:55.635649 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:55.636727 (+ 1078us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:55.638561 (+ 1834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:55.639205 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:55.639298 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:43:55.641697 (+ 2399us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:43:55.641704 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:43:55.641709 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:43:55.641780 (+ 71us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:43:55.645259 (+ 3479us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:43:55.646021 (+ 762us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:43:55.646027 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:43:55.646030 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:43:55.646178 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:43:55.646581 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:43:55.646586 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:43:55.646589 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:43:55.646937 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:43:55.647094 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0619 16:43:55.647500 (+ 406us) server_negotiation.cc:299] Negotiation successful
0619 16:43:55.647733 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260619 16:43:55.648121 7096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:55.633347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46817 (local address 127.0.0.1:46064)
0619 16:43:55.633967 (+ 620us) negotiation.cc:107] Waiting for socket to connect
0619 16:43:55.633990 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0619 16:43:55.634189 (+ 199us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:43:55.634724 (+ 535us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:43:55.634729 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:43:55.635224 (+ 495us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:43:55.635474 (+ 250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:43:55.635486 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:43:55.636933 (+ 1447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:43:55.636942 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:43:55.638396 (+ 1454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:43:55.638414 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:43:55.639529 (+ 1115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:43:55.639534 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:43:55.639708 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:43:55.640398 (+ 690us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:43:55.640421 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:43:55.641496 (+ 1075us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:43:55.645382 (+ 3886us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:43:55.645387 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:43:55.645392 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:43:55.645904 (+ 512us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:43:55.646285 (+ 381us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:43:55.646289 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:43:55.646293 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:43:55.646468 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:43:55.647061 (+ 593us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:43:55.647067 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:43:55.647232 (+ 165us) client_negotiation.cc:769] Sending connection context
0619 16:43:55.647378 (+ 146us) client_negotiation.cc:240] Negotiation successful
0619 16:43:55.647663 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":494,"thread_start_us":375,"threads_started":1}
I20260619 16:43:55.649413 6547 tablet_copy_service.cc:140] P 5ba5d12f1093420e96661d1c478add9b: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 85d1444e474f4e189618fb22f5a42908 ({username='oryx', principal='oryx/127.0.97.61@KRBTEST.COM'} at 127.0.0.1:46064)
I20260619 16:43:55.649639 6547 tablet_copy_service.cc:161] P 5ba5d12f1093420e96661d1c478add9b: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 85d1444e474f4e189618fb22f5a42908 at {username='oryx', principal='oryx/127.0.97.61@KRBTEST.COM'} at 127.0.0.1:46064: session id = 85d1444e474f4e189618fb22f5a42908-00000000000000000000000000000000
I20260619 16:43:55.653877 6547 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 5ba5d12f1093420e96661d1c478add9b: Tablet Copy: opened 0 blocks and 1 log segments
I20260619 16:43:55.656035 6988 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260619 16:43:55.663168 6988 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 85d1444e474f4e189618fb22f5a42908: tablet copy: Starting download of 0 data blocks...
I20260619 16:43:55.663491 6988 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 85d1444e474f4e189618fb22f5a42908: tablet copy: Starting download of 1 WAL segments...
I20260619 16:43:55.666531 6988 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 85d1444e474f4e189618fb22f5a42908: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260619 16:43:55.671135 6547 tablet_copy_service.cc:342] P 5ba5d12f1093420e96661d1c478add9b: Request end of tablet copy session 85d1444e474f4e189618fb22f5a42908-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.97.61@KRBTEST.COM'} at 127.0.0.1:46064
I20260619 16:43:55.671325 6547 tablet_copy_service.cc:433] P 5ba5d12f1093420e96661d1c478add9b: ending tablet copy session 85d1444e474f4e189618fb22f5a42908-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 85d1444e474f4e189618fb22f5a42908
I20260619 16:43:55.682091 6988 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:43:55.684406 7097 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:43:55.690462 7098 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:43:55.693967 7100 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:43:55.695062 6988 server_base.cc:1034] running on GCE node
I20260619 16:43:55.695343 6988 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:43:55.695863 6988 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:43:55.697080 6988 hybrid_clock.cc:648] HybridClock initialized: now 1781887435697025 us; error 34 us; skew 500 ppm
I20260619 16:43:55.697270 6988 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:43:55 dist-test-slave-wvz7 krb5kdc[6482](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887435, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:43:55.699023 6988 init.cc:377] Logged in from keytab as oryx/127.0.97.61@KRBTEST.COM (short username oryx)
I20260619 16:43:55.702473 6988 webserver.cc:460] Webserver started at http://127.0.97.61:44321/ using document root <none> and password file <none>
I20260619 16:43:55.702755 6988 fs_manager.cc:359] Metadata directory not provided
I20260619 16:43:55.702826 6988 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:43:55.705173 6988 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.001s
I20260619 16:43:55.706756 7107 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:43:55.707125 6988 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 16:43:55.707249 6988 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887385903558-388-0/minicluster-data/master-1/data,/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887385903558-388-0/minicluster-data/master-1/wal
uuid: "85d1444e474f4e189618fb22f5a42908"
format_stamp: "Formatted at 2026-06-19 16:43:55 on dist-test-slave-wvz7"
server_key: "c67ccda032cbde5cc15fec7ba65504f1"
server_key_iv: "265dcd58e69572ada7ceccf68de61e88"
server_key_version: "encryptionkey@0"
I20260619 16:43:55.707397 6988 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887385903558-388-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887385903558-388-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887385903558-388-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:43:55.719812 6988 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:43:55.722712 6988 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:43:55.750926 6988 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.61:42337
I20260619 16:43:55.751766 6988 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887385903558-388-0/minicluster-data/master-1/data/info.pb
I20260619 16:43:55.750926 7158 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.61:42337 every 8 connection(s)
I20260619 16:43:55.757819 7159 sys_catalog.cc:263] Verifying existing consensus state
I20260619 16:43:55.760695 7159 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 85d1444e474f4e189618fb22f5a42908: Bootstrap starting.
I20260619 16:43:55.772622 7159 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 85d1444e474f4e189618fb22f5a42908: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 16:43:55.773028 7159 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 85d1444e474f4e189618fb22f5a42908: Bootstrap complete.
I20260619 16:43:55.773936 7159 raft_consensus.cc:348] T 00000000000000000000000000000000 P 85d1444e474f4e189618fb22f5a42908 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "5ba5d12f1093420e96661d1c478add9b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46817 } } peers { permanent_uuid: "85d1444e474f4e189618fb22f5a42908" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 42337 } attrs { promote: true } }
I20260619 16:43:55.774132 7159 raft_consensus.cc:729] T 00000000000000000000000000000000 P 85d1444e474f4e189618fb22f5a42908 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 85d1444e474f4e189618fb22f5a42908, State: Initialized, Role: LEARNER
I20260619 16:43:55.774358 7159 consensus_queue.cc:260] T 00000000000000000000000000000000 P 85d1444e474f4e189618fb22f5a42908 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "5ba5d12f1093420e96661d1c478add9b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46817 } } peers { permanent_uuid: "85d1444e474f4e189618fb22f5a42908" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 42337 } attrs { promote: true } }
I20260619 16:43:55.775369 7162 sys_catalog.cc:455] T 00000000000000000000000000000000 P 85d1444e474f4e189618fb22f5a42908 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "5ba5d12f1093420e96661d1c478add9b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46817 } } peers { permanent_uuid: "85d1444e474f4e189618fb22f5a42908" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 42337 } attrs { promote: true } } }
I20260619 16:43:55.775607 7159 sys_catalog.cc:564] T 00000000000000000000000000000000 P 85d1444e474f4e189618fb22f5a42908 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:43:55.775774 7162 sys_catalog.cc:458] T 00000000000000000000000000000000 P 85d1444e474f4e189618fb22f5a42908 [sys.catalog]: This master's current role is: LEARNER
I20260619 16:43:55.782837 7173 catalog_manager.cc:1259] Loaded cluster ID: dc389b63eb214c9bb142cff4aef89cd5
I20260619 16:43:55.782924 7173 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 85d1444e474f4e189618fb22f5a42908: loading cluster ID for follower catalog manager: success
I20260619 16:43:55.785209 7173 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 85d1444e474f4e189618fb22f5a42908: acquiring CA information for follower catalog manager: success
I20260619 16:43:55.787022 7173 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 85d1444e474f4e189618fb22f5a42908: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260619 16:43:55.992617 7087 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:43:55.992928 7087 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:43:55.993088 7087 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:43:56.000342 7087 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:43:56.000469 7087 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:43:56.000543 7087 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:43:56.000633 7087 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:43:56.000689 7087 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:43:56.005131 6983 heartbeater.cc:503] Master 127.0.97.62:46817 was elected leader, sending a full tablet report...
I20260619 16:43:56.011502 7087 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44537
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887385903558-388-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887385903558-388-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:46817,127.0.97.61:42337,127.0.97.60:35535
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887385903558-388-0/krb5kdc/oryx_127.0.97.60.keytab
--principal=oryx/127.0.97.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:35535
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887385903558-388-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.97.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887385903558-388-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:43:56.012856 7087 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:43:56.014201 7087 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:43:56.026209 7179 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:43:56.026304 7178 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:43:56.026257 7181 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:43:56.026965 7087 server_base.cc:1034] running on GCE node
I20260619 16:43:56.027755 7087 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:43:56.029212 7087 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:43:56.030433 7087 hybrid_clock.cc:648] HybridClock initialized: now 1781887436030384 us; error 46 us; skew 500 ppm
I20260619 16:43:56.030827 7087 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:43:56 dist-test-slave-wvz7 krb5kdc[6482](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887436, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:43:56.034066 7087 init.cc:377] Logged in from keytab as oryx/127.0.97.60@KRBTEST.COM (short username oryx)
I20260619 16:43:56.035614 7087 webserver.cc:460] Webserver started at http://127.0.97.60:33443/ using document root <none> and password file <none>
I20260619 16:43:56.036290 7087 fs_manager.cc:359] Metadata directory not provided
I20260619 16:43:56.036420 7087 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:43:56.036690 7087 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:43:56.039613 7087 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887385903558-388-0/minicluster-data/master-2/data/instance:
uuid: "2e547db70ef542a88009307c8181f39d"
format_stamp: "Formatted at 2026-06-19 16:43:56 on dist-test-slave-wvz7"
server_key: "569c9749205ec3d458c9d0ceeaa154b8"
server_key_iv: "9d2081e8d2c49a5feb94ac45a3370f39"
server_key_version: "encryptionkey@0"
I20260619 16:43:56.040328 7087 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887385903558-388-0/minicluster-data/master-2/wal/instance:
uuid: "2e547db70ef542a88009307c8181f39d"
format_stamp: "Formatted at 2026-06-19 16:43:56 on dist-test-slave-wvz7"
server_key: "569c9749205ec3d458c9d0ceeaa154b8"
server_key_iv: "9d2081e8d2c49a5feb94ac45a3370f39"
server_key_version: "encryptionkey@0"
I20260619 16:43:56.045598 7087 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260619 16:43:56.049350 7188 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:43:56.050681 7087 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 16:43:56.050824 7087 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887385903558-388-0/minicluster-data/master-2/data,/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887385903558-388-0/minicluster-data/master-2/wal
uuid: "2e547db70ef542a88009307c8181f39d"
format_stamp: "Formatted at 2026-06-19 16:43:56 on dist-test-slave-wvz7"
server_key: "569c9749205ec3d458c9d0ceeaa154b8"
server_key_iv: "9d2081e8d2c49a5feb94ac45a3370f39"
server_key_version: "encryptionkey@0"
I20260619 16:43:56.050984 7087 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887385903558-388-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887385903558-388-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887385903558-388-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:43:56.064751 7087 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:43:56.068706 7087 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:43:56.068953 7087 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:43:56.086962 7087 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.60:35535
I20260619 16:43:56.086982 7239 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.60:35535 every 8 connection(s)
I20260619 16:43:56.088228 7087 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887385903558-388-0/minicluster-data/master-2/data/info.pb
I20260619 16:43:56.091784 7240 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:43:56.091843 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu as pid 7087
I20260619 16:43:56.101222 7240 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46817 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 16:43:56 dist-test-slave-wvz7 krb5kdc[6482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887433, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.60@KRBTEST.COM
Jun 19 16:43:56 dist-test-slave-wvz7 krb5kdc[6482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887436, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.60@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260619 16:43:56.114992 7243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:56.093704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55058 (local address 127.0.97.60:35535)
0619 16:43:56.094952 (+ 1248us) server_negotiation.cc:206] Beginning negotiation
0619 16:43:56.094967 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 16:43:56.095013 (+ 46us) server_negotiation.cc:407] Connection header received
0619 16:43:56.096601 (+ 1588us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:43:56.096626 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:43:56.097101 (+ 475us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:43:56.097523 (+ 422us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:43:56.098769 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:56.100634 (+ 1865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:56.101877 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:56.102789 (+ 912us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:56.102941 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:43:56.107174 (+ 4233us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:43:56.107183 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:43:56.107197 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:43:56.107241 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:43:56.110926 (+ 3685us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:43:56.111652 (+ 726us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:43:56.111660 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:43:56.111668 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:43:56.111824 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:43:56.112169 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:43:56.112174 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:43:56.112178 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:43:56.112797 (+ 619us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:43:56.113085 (+ 288us) server_negotiation.cc:1035] Waiting for connection context
0619 16:43:56.113433 (+ 348us) server_negotiation.cc:299] Negotiation successful
0619 16:43:56.113953 (+ 520us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1009,"thread_start_us":309,"threads_started":1}
I20260619 16:43:56.119259 6718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:56.103927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46066 (local address 127.0.97.62:46817)
0619 16:43:56.104094 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0619 16:43:56.104098 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:43:56.106237 (+ 2139us) server_negotiation.cc:407] Connection header received
0619 16:43:56.106369 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:43:56.106374 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:43:56.106449 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:43:56.106548 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:43:56.108083 (+ 1535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:56.108829 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:56.110726 (+ 1897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:56.111185 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:56.111305 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:43:56.114381 (+ 3076us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:43:56.114386 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:43:56.114389 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:43:56.114419 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:43:56.116848 (+ 2429us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:43:56.117538 (+ 690us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:43:56.117544 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:43:56.117547 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:43:56.117656 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:43:56.117997 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:43:56.118002 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:43:56.118005 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:43:56.118244 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:43:56.118364 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0619 16:43:56.118772 (+ 408us) server_negotiation.cc:299] Negotiation successful
0619 16:43:56.118954 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260619 16:43:56.119310 7244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:56.103790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46817 (local address 127.0.0.1:46066)
0619 16:43:56.106062 (+ 2272us) negotiation.cc:107] Waiting for socket to connect
0619 16:43:56.106103 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0619 16:43:56.106259 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:43:56.106603 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:43:56.106607 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:43:56.107218 (+ 611us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:43:56.107957 (+ 739us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:43:56.107970 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:43:56.108944 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:43:56.108949 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:43:56.110581 (+ 1632us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:43:56.110595 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:43:56.111347 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:43:56.111352 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:43:56.111461 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:43:56.111953 (+ 492us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:43:56.111973 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:43:56.114209 (+ 2236us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:43:56.116964 (+ 2755us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:43:56.116973 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:43:56.116980 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:43:56.117439 (+ 459us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:43:56.117768 (+ 329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:43:56.117773 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:43:56.117777 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:43:56.117895 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:43:56.118353 (+ 458us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:43:56.118359 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:43:56.118502 (+ 143us) client_negotiation.cc:769] Sending connection context
0619 16:43:56.118646 (+ 144us) client_negotiation.cc:240] Negotiation successful
0619 16:43:56.118926 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2033,"thread_start_us":1850,"threads_started":1}
I20260619 16:43:56.120509 7240 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42337 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 16:43:56 dist-test-slave-wvz7 krb5kdc[6482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887436, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.60@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
I20260619 16:43:56.132164 7247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:56.125632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:42337 (local address 127.0.0.1:42140)
0619 16:43:56.126050 (+ 418us) negotiation.cc:107] Waiting for socket to connect
0619 16:43:56.126065 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 16:43:56.126157 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:43:56.126736 (+ 579us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:43:56.126741 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:43:56.126767 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:43:56.126950 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:43:56.126960 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:43:56.128333 (+ 1373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:43:56.128337 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:43:56.130056 (+ 1719us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:43:56.130072 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:43:56.131388 (+ 1316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:43:56.131394 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:43:56.131570 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:43:56.131622 (+ 52us) client_negotiation.cc:769] Sending connection context
0619 16:43:56.131785 (+ 163us) client_negotiation.cc:240] Negotiation successful
0619 16:43:56.131908 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":321,"thread_start_us":208,"threads_started":1}
I20260619 16:43:56.132251 7248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:56.125766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42140 (local address 127.0.97.61:42337)
0619 16:43:56.126465 (+ 699us) server_negotiation.cc:206] Beginning negotiation
0619 16:43:56.126469 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:43:56.126482 (+ 13us) server_negotiation.cc:407] Connection header received
0619 16:43:56.126527 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:43:56.126531 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:43:56.126585 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:43:56.126674 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:43:56.127094 (+ 420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:56.128188 (+ 1094us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:56.130209 (+ 2021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:56.131238 (+ 1029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:56.131337 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:43:56.131509 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0619 16:43:56.131815 (+ 306us) server_negotiation.cc:299] Negotiation successful
0619 16:43:56.131917 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":601,"thread_start_us":480,"threads_started":1}
I20260619 16:43:56.133279 7134 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 85d1444e474f4e189618fb22f5a42908 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20260619 16:43:56.133932 7162 sys_catalog.cc:455] T 00000000000000000000000000000000 P 85d1444e474f4e189618fb22f5a42908 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5ba5d12f1093420e96661d1c478add9b. Latest consensus state: current_term: 1 leader_uuid: "5ba5d12f1093420e96661d1c478add9b" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "5ba5d12f1093420e96661d1c478add9b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46817 } } peers { permanent_uuid: "85d1444e474f4e189618fb22f5a42908" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 42337 } attrs { promote: true } } }
I20260619 16:43:56.134195 7162 sys_catalog.cc:458] T 00000000000000000000000000000000 P 85d1444e474f4e189618fb22f5a42908 [sys.catalog]: This master's current role is: LEARNER
I20260619 16:43:56.134431 7244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:56.121039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:42337 (local address 127.0.0.1:42128)
0619 16:43:56.121211 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0619 16:43:56.121228 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 16:43:56.121320 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:43:56.121902 (+ 582us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:43:56.121906 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:43:56.122209 (+ 303us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:43:56.122328 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:43:56.122335 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:43:56.123485 (+ 1150us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:43:56.123489 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:43:56.124797 (+ 1308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:43:56.124808 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:43:56.125526 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:43:56.125531 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:43:56.125685 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:43:56.126379 (+ 694us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:43:56.126395 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:43:56.128582 (+ 2187us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:43:56.131982 (+ 3400us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:43:56.131987 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:43:56.131992 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:43:56.132482 (+ 490us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:43:56.132873 (+ 391us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:43:56.132878 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:43:56.132882 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:43:56.133002 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:43:56.133533 (+ 531us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:43:56.133538 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:43:56.133704 (+ 166us) client_negotiation.cc:769] Sending connection context
0619 16:43:56.133878 (+ 174us) client_negotiation.cc:240] Negotiation successful
0619 16:43:56.134131 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42}
I20260619 16:43:56.134508 7245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:56.121191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42128 (local address 127.0.97.61:42337)
0619 16:43:56.121621 (+ 430us) server_negotiation.cc:206] Beginning negotiation
0619 16:43:56.121626 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:43:56.121641 (+ 15us) server_negotiation.cc:407] Connection header received
0619 16:43:56.121686 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:43:56.121690 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:43:56.121781 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:43:56.121858 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:43:56.122431 (+ 573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:56.123372 (+ 941us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:56.124946 (+ 1574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:56.125384 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:56.125501 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:43:56.128749 (+ 3248us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:43:56.128787 (+ 38us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:43:56.128792 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:43:56.128828 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:43:56.131876 (+ 3048us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:43:56.132587 (+ 711us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:43:56.132591 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:43:56.132593 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:43:56.132729 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:43:56.133095 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:43:56.133099 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:43:56.133107 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:43:56.133420 (+ 313us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:43:56.133570 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0619 16:43:56.133978 (+ 408us) server_negotiation.cc:299] Negotiation successful
0619 16:43:56.134158 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":326,"thread_start_us":213,"threads_started":1}
I20260619 16:43:56.135385 7240 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35535 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 16:43:56 dist-test-slave-wvz7 krb5kdc[6482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887436, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.60@KRBTEST.COM for oryx/127.0.97.60@KRBTEST.COM
I20260619 16:43:56.146469 7244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:56.135918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:35535 (local address 127.0.0.1:55074)
0619 16:43:56.136513 (+ 595us) negotiation.cc:107] Waiting for socket to connect
0619 16:43:56.136530 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 16:43:56.136613 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:43:56.136855 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:43:56.136858 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:43:56.137134 (+ 276us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:43:56.137253 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:43:56.137260 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:43:56.138161 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:43:56.138165 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:43:56.138963 (+ 798us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:43:56.138973 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:43:56.139554 (+ 581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:43:56.139557 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:43:56.139740 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:43:56.140164 (+ 424us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:43:56.140179 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:43:56.142078 (+ 1899us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:43:56.144455 (+ 2377us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:43:56.144458 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:43:56.144461 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:43:56.144813 (+ 352us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:43:56.145102 (+ 289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:43:56.145107 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:43:56.145111 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:43:56.145260 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:43:56.145717 (+ 457us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:43:56.145720 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:43:56.145907 (+ 187us) client_negotiation.cc:769] Sending connection context
0619 16:43:56.146013 (+ 106us) client_negotiation.cc:240] Negotiation successful
0619 16:43:56.146189 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48}
I20260619 16:43:56.146620 7243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:56.136488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55074 (local address 127.0.97.60:35535)
0619 16:43:56.136612 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0619 16:43:56.136615 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 16:43:56.136629 (+ 14us) server_negotiation.cc:407] Connection header received
0619 16:43:56.136707 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:43:56.136710 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:43:56.136764 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:43:56.136832 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:43:56.137350 (+ 518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:56.138031 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:56.139074 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:56.139434 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:56.139519 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:43:56.142212 (+ 2693us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:43:56.142216 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:43:56.142219 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:43:56.142244 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:43:56.144346 (+ 2102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:43:56.144901 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:43:56.144905 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:43:56.144907 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:43:56.144999 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:43:56.145346 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:43:56.145349 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:43:56.145351 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:43:56.145585 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:43:56.145683 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0619 16:43:56.146086 (+ 403us) server_negotiation.cc:299] Negotiation successful
0619 16:43:56.146341 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":70,"server-negotiator.queue_time_us":44}
I20260619 16:43:56.150827 7240 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2e547db70ef542a88009307c8181f39d: Bootstrap starting.
I20260619 16:43:56.153728 7240 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2e547db70ef542a88009307c8181f39d: Neither blocks nor log segments found. Creating new log.
I20260619 16:43:56.154764 7240 log.cc:824] T 00000000000000000000000000000000 P 2e547db70ef542a88009307c8181f39d: Log is configured to *not* fsync() on all Append() calls
I20260619 16:43:56.157573 7240 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2e547db70ef542a88009307c8181f39d: No bootstrap required, opened a new log
I20260619 16:43:56.164093 7240 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2e547db70ef542a88009307c8181f39d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ba5d12f1093420e96661d1c478add9b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46817 } } peers { permanent_uuid: "85d1444e474f4e189618fb22f5a42908" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42337 } } peers { permanent_uuid: "2e547db70ef542a88009307c8181f39d" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35535 } }
I20260619 16:43:56.164441 7240 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2e547db70ef542a88009307c8181f39d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:43:56.164567 7240 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2e547db70ef542a88009307c8181f39d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2e547db70ef542a88009307c8181f39d, State: Initialized, Role: FOLLOWER
I20260619 16:43:56.165096 7240 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2e547db70ef542a88009307c8181f39d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ba5d12f1093420e96661d1c478add9b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46817 } } peers { permanent_uuid: "85d1444e474f4e189618fb22f5a42908" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42337 } } peers { permanent_uuid: "2e547db70ef542a88009307c8181f39d" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35535 } }
I20260619 16:43:56.165891 7250 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2e547db70ef542a88009307c8181f39d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ba5d12f1093420e96661d1c478add9b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46817 } } peers { permanent_uuid: "85d1444e474f4e189618fb22f5a42908" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42337 } } peers { permanent_uuid: "2e547db70ef542a88009307c8181f39d" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35535 } } }
I20260619 16:43:56.166157 7250 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2e547db70ef542a88009307c8181f39d [sys.catalog]: This master's current role is: FOLLOWER
I20260619 16:43:56.166838 7240 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2e547db70ef542a88009307c8181f39d [sys.catalog]: configured and running, proceeding with master startup.
W20260619 16:43:56.175206 7261 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 2e547db70ef542a88009307c8181f39d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 16:43:56.175345 7261 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 16:43:56.179719 7087 master_runner.cc:428] Detected that this master 2e547db70ef542a88009307c8181f39d is joining an existing cluster
I20260619 16:43:56.179832 7087 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.97.60:35535
I20260619 16:43:56.200677 6718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:56.183240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46078 (local address 127.0.97.62:46817)
0619 16:43:56.183414 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0619 16:43:56.183420 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:43:56.184308 (+ 888us) server_negotiation.cc:407] Connection header received
0619 16:43:56.184553 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:43:56.184559 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:43:56.184640 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:43:56.184717 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:43:56.186839 (+ 2122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:56.188051 (+ 1212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:56.190254 (+ 2203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:56.190996 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:56.191095 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:43:56.193991 (+ 2896us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:43:56.193999 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:43:56.194005 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:43:56.194049 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:43:56.197431 (+ 3382us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:43:56.198273 (+ 842us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:43:56.198279 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:43:56.198283 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:43:56.198434 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:43:56.198914 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:43:56.198918 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:43:56.198922 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:43:56.199254 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:43:56.199402 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0619 16:43:56.200051 (+ 649us) server_negotiation.cc:299] Negotiation successful
0619 16:43:56.200282 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260619 16:43:56.201759 7267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:56.183429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46817 (local address 127.0.0.1:46078)
0619 16:43:56.184220 (+ 791us) negotiation.cc:107] Waiting for socket to connect
0619 16:43:56.184237 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 16:43:56.184407 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:43:56.184945 (+ 538us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:43:56.184950 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:43:56.185638 (+ 688us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:43:56.186107 (+ 469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:43:56.186122 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:43:56.188234 (+ 2112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:43:56.188240 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:43:56.190076 (+ 1836us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:43:56.190092 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:43:56.191389 (+ 1297us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:43:56.191395 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:43:56.191610 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:43:56.192393 (+ 783us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:43:56.192420 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:43:56.193673 (+ 1253us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:43:56.197610 (+ 3937us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:43:56.197616 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:43:56.197620 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:43:56.198143 (+ 523us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:43:56.198575 (+ 432us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:43:56.198584 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:43:56.198588 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:43:56.198787 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:43:56.199480 (+ 693us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:43:56.199485 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:43:56.199723 (+ 238us) client_negotiation.cc:769] Sending connection context
0619 16:43:56.200927 (+ 1204us) client_negotiation.cc:240] Negotiation successful
0619 16:43:56.201241 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":509,"thread_start_us":365,"threads_started":1}
I20260619 16:43:56.206632 7245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:56.183951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42144 (local address 127.0.97.61:42337)
0619 16:43:56.184101 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0619 16:43:56.184106 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:43:56.191705 (+ 7599us) server_negotiation.cc:407] Connection header received
0619 16:43:56.191858 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:43:56.191864 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:43:56.191963 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:43:56.192052 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:43:56.193501 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:56.194617 (+ 1116us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:56.196651 (+ 2034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:56.197366 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:56.197481 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:43:56.199784 (+ 2303us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:43:56.199790 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:43:56.199795 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:43:56.199834 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:43:56.203066 (+ 3232us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:43:56.204108 (+ 1042us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:43:56.204113 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:43:56.204117 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:43:56.204274 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:43:56.204729 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:43:56.204735 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:43:56.204738 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:43:56.205060 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:43:56.205185 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0619 16:43:56.205809 (+ 624us) server_negotiation.cc:299] Negotiation successful
0619 16:43:56.206015 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260619 16:43:56.209884 7268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:56.183506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:42337 (local address 127.0.0.1:42144)
0619 16:43:56.191602 (+ 8096us) negotiation.cc:107] Waiting for socket to connect
0619 16:43:56.191618 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 16:43:56.191747 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:43:56.192224 (+ 477us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:43:56.192229 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:43:56.192983 (+ 754us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:43:56.193316 (+ 333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:43:56.193331 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:43:56.194811 (+ 1480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:43:56.194817 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:43:56.196491 (+ 1674us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:43:56.196505 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:43:56.197756 (+ 1251us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:43:56.197762 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:43:56.197922 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:43:56.198502 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:43:56.198524 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:43:56.199585 (+ 1061us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:43:56.203509 (+ 3924us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:43:56.203514 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:43:56.203518 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:43:56.203980 (+ 462us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:43:56.204418 (+ 438us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:43:56.204423 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:43:56.204427 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:43:56.204603 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:43:56.205289 (+ 686us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:43:56.205293 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:43:56.205510 (+ 217us) client_negotiation.cc:769] Sending connection context
0619 16:43:56.207012 (+ 1502us) client_negotiation.cc:240] Negotiation successful
0619 16:43:56.208081 (+ 1069us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7992,"thread_start_us":7813,"threads_started":1}
I20260619 16:43:56.216353 7267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:56.184543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:35535 (local address 127.0.0.1:55088)
0619 16:43:56.201969 (+ 17426us) negotiation.cc:107] Waiting for socket to connect
0619 16:43:56.201986 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 16:43:56.202115 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:43:56.202505 (+ 390us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:43:56.202509 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:43:56.203005 (+ 496us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:43:56.203202 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:43:56.203212 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:43:56.204876 (+ 1664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:43:56.204881 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:43:56.206186 (+ 1305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:43:56.206199 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:43:56.207144 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:43:56.207150 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:43:56.207345 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:43:56.208028 (+ 683us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:43:56.208053 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:43:56.210668 (+ 2615us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:43:56.214024 (+ 3356us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:43:56.214028 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:43:56.214031 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:43:56.214500 (+ 469us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:43:56.214826 (+ 326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:43:56.214831 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:43:56.214835 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:43:56.214941 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:43:56.215428 (+ 487us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:43:56.215432 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:43:56.215552 (+ 120us) client_negotiation.cc:769] Sending connection context
0619 16:43:56.215698 (+ 146us) client_negotiation.cc:240] Negotiation successful
0619 16:43:56.215982 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17342,"thread_start_us":18908,"threads_started":1}
Jun 19 16:43:56 dist-test-slave-wvz7 krb5kdc[6482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887433, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.62@KRBTEST.COM for oryx/127.0.97.60@KRBTEST.COM
I20260619 16:43:56.216967 7243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:56.184296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55088 (local address 127.0.97.60:35535)
0619 16:43:56.184450 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0619 16:43:56.184454 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:43:56.202037 (+ 17583us) server_negotiation.cc:407] Connection header received
0619 16:43:56.202223 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:43:56.202228 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:43:56.202286 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:43:56.202343 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:43:56.203667 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:56.204665 (+ 998us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:56.206307 (+ 1642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:56.207013 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:56.207119 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:43:56.210849 (+ 3730us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:43:56.210854 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:43:56.210858 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:43:56.210886 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:43:56.213903 (+ 3017us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:43:56.214601 (+ 698us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:43:56.214606 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:43:56.214610 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:43:56.214735 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:43:56.215053 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:43:56.215058 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:43:56.215061 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:43:56.215325 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:43:56.215464 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0619 16:43:56.215821 (+ 357us) server_negotiation.cc:299] Negotiation successful
0619 16:43:56.216631 (+ 810us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":627,"server-negotiator.queue_time_us":54}
I20260619 16:43:56.220253 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu pid 6588
I20260619 16:43:56.224009 7247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:56.208628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:35535 (local address 127.0.0.1:55092)
0619 16:43:56.208805 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0619 16:43:56.208824 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 16:43:56.208920 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:43:56.209597 (+ 677us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:43:56.209602 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:43:56.210042 (+ 440us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:43:56.210216 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:43:56.210228 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:43:56.211780 (+ 1552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:43:56.211786 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:43:56.213024 (+ 1238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:43:56.213039 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:43:56.214031 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:43:56.214037 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:43:56.214199 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:43:56.214756 (+ 557us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:43:56.214778 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:43:56.217292 (+ 2514us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:43:56.221220 (+ 3928us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:43:56.221225 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:43:56.221229 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:43:56.221817 (+ 588us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:43:56.222198 (+ 381us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:43:56.222203 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:43:56.222207 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:43:56.222369 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:43:56.222970 (+ 601us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:43:56.222975 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:43:56.223170 (+ 195us) client_negotiation.cc:769] Sending connection context
0619 16:43:56.223326 (+ 156us) client_negotiation.cc:240] Negotiation successful
0619 16:43:56.223620 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47}
I20260619 16:43:56.224067 7270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:56.208817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55092 (local address 127.0.97.60:35535)
0619 16:43:56.209308 (+ 491us) server_negotiation.cc:206] Beginning negotiation
0619 16:43:56.209312 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:43:56.209328 (+ 16us) server_negotiation.cc:407] Connection header received
0619 16:43:56.209394 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:43:56.209397 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:43:56.209452 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:43:56.209527 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:43:56.210416 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:56.211653 (+ 1237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:56.213172 (+ 1519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:56.213885 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:56.214009 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:43:56.217577 (+ 3568us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:43:56.217585 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:43:56.217590 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:43:56.217628 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:43:56.221059 (+ 3431us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:43:56.221933 (+ 874us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:43:56.221938 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:43:56.221942 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:43:56.222084 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:43:56.222490 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:43:56.222494 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:43:56.222498 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:43:56.222856 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:43:56.223010 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0619 16:43:56.223425 (+ 415us) server_negotiation.cc:299] Negotiation successful
0619 16:43:56.223650 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":403,"spinlock_wait_cycles":35200,"thread_start_us":278,"threads_started":1}
I20260619 16:43:56.225934 6527 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.97.60:35535: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "2e547db70ef542a88009307c8181f39d"
member_type: NON_VOTER
last_known_addr {
host: "127.0.97.60"
port: 35535
}
attrs {
promote: true
}
}
dest_uuid: "5ba5d12f1093420e96661d1c478add9b"
cas_config_opid_index: 5
I20260619 16:43:56.226922 6527 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5ba5d12f1093420e96661d1c478add9b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "5ba5d12f1093420e96661d1c478add9b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46817 } } peers { permanent_uuid: "85d1444e474f4e189618fb22f5a42908" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 42337 } attrs { promote: true } } peers { permanent_uuid: "2e547db70ef542a88009307c8181f39d" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 35535 } attrs { promote: true } }
I20260619 16:43:56.229532 7246 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 5ba5d12f1093420e96661d1c478add9b [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 2e547db70ef542a88009307c8181f39d (127.0.97.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "5ba5d12f1093420e96661d1c478add9b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46817 } } peers { permanent_uuid: "85d1444e474f4e189618fb22f5a42908" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 42337 } attrs { promote: true } } peers { permanent_uuid: "2e547db70ef542a88009307c8181f39d" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 35535 } attrs { promote: true } } }
I20260619 16:43:56.233417 7246 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.97.60:35535
I20260619 16:43:56.234650 7134 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 85d1444e474f4e189618fb22f5a42908 [term 1 LEARNER]: Refusing update from remote peer 5ba5d12f1093420e96661d1c478add9b: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260619 16:43:56.235021 7215 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2e547db70ef542a88009307c8181f39d [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:43:56.237466 7272 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 5ba5d12f1093420e96661d1c478add9b [LEADER]: Connected to new peer: Peer: permanent_uuid: "85d1444e474f4e189618fb22f5a42908" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 42337 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260619 16:43:56.238634 7215 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 2e547db70ef542a88009307c8181f39d [term 1 FOLLOWER]: Refusing update from remote peer 5ba5d12f1093420e96661d1c478add9b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260619 16:43:56.240849 7246 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 5ba5d12f1093420e96661d1c478add9b [LEADER]: Connected to new peer: Peer: permanent_uuid: "2e547db70ef542a88009307c8181f39d" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 35535 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:43:56.241218 7087 master.cc:537] Master@127.0.97.60:35535 shutting down...
I20260619 16:43:56.241096 7273 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5ba5d12f1093420e96661d1c478add9b [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "5ba5d12f1093420e96661d1c478add9b" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "5ba5d12f1093420e96661d1c478add9b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46817 } } peers { permanent_uuid: "85d1444e474f4e189618fb22f5a42908" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 42337 } attrs { promote: true } } peers { permanent_uuid: "2e547db70ef542a88009307c8181f39d" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 35535 } attrs { promote: true } } }
I20260619 16:43:56.242244 7273 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5ba5d12f1093420e96661d1c478add9b [sys.catalog]: This master's current role is: LEADER
I20260619 16:43:56.252308 7087 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 2e547db70ef542a88009307c8181f39d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 16:43:56.252729 7087 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 2e547db70ef542a88009307c8181f39d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:43:56.252982 7087 tablet_replica.cc:330] T 00000000000000000000000000000000 P 2e547db70ef542a88009307c8181f39d: stopping tablet replica
Jun 19 16:43:56 dist-test-slave-wvz7 krb5kdc[6482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887433, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.1@KRBTEST.COM
W20260619 16:43:56.259330 6514 consensus_peers.cc:487] T 00000000000000000000000000000000 P 5ba5d12f1093420e96661d1c478add9b -> Peer 2e547db70ef542a88009307c8181f39d (127.0.97.60:35535): Couldn't send request to peer 2e547db70ef542a88009307c8181f39d. Status: Network error: recv got EOF from unknown peer (error 108). This is attempt 1: this message will repeat every 5th retry.
I20260619 16:43:56.261123 7087 master.cc:552] Master@127.0.97.60:35535 shutdown complete.
I20260619 16:43:56.261207 7087 master_runner.cc:305] Clearing existing system tablet
I20260619 16:43:56.260515 7134 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 85d1444e474f4e189618fb22f5a42908 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 2e547db70ef542a88009307c8181f39d (127.0.97.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "5ba5d12f1093420e96661d1c478add9b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46817 } } peers { permanent_uuid: "85d1444e474f4e189618fb22f5a42908" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 42337 } attrs { promote: true } } peers { permanent_uuid: "2e547db70ef542a88009307c8181f39d" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 35535 } attrs { promote: true } } }
I20260619 16:43:56.261582 7087 fs_manager.cc:359] Metadata directory not provided
I20260619 16:43:56.261683 7087 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:43:56.265446 7162 sys_catalog.cc:455] T 00000000000000000000000000000000 P 85d1444e474f4e189618fb22f5a42908 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5ba5d12f1093420e96661d1c478add9b" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "5ba5d12f1093420e96661d1c478add9b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46817 } } peers { permanent_uuid: "85d1444e474f4e189618fb22f5a42908" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 42337 } attrs { promote: true } } peers { permanent_uuid: "2e547db70ef542a88009307c8181f39d" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 35535 } attrs { promote: true } } }
I20260619 16:43:56.265785 7162 sys_catalog.cc:458] T 00000000000000000000000000000000 P 85d1444e474f4e189618fb22f5a42908 [sys.catalog]: This master's current role is: LEARNER
I20260619 16:43:56.267251 7274 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 5ba5d12f1093420e96661d1c478add9b: attempting to promote NON_VOTER 85d1444e474f4e189618fb22f5a42908 to VOTER
I20260619 16:43:56.268173 7274 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5ba5d12f1093420e96661d1c478add9b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "5ba5d12f1093420e96661d1c478add9b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46817 } } peers { permanent_uuid: "85d1444e474f4e189618fb22f5a42908" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42337 } attrs { promote: false } } peers { permanent_uuid: "2e547db70ef542a88009307c8181f39d" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 35535 } attrs { promote: true } }
I20260619 16:43:56.269424 7271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:56.221210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33742 (local address 127.0.97.1:40321)
0619 16:43:56.237075 (+ 15865us) server_negotiation.cc:206] Beginning negotiation
0619 16:43:56.237085 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 16:43:56.237120 (+ 35us) server_negotiation.cc:407] Connection header received
0619 16:43:56.237194 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:43:56.237204 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:43:56.237503 (+ 299us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:43:56.239874 (+ 2371us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:43:56.240100 (+ 226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:56.244580 (+ 4480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:56.248796 (+ 4216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:56.249613 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:56.259265 (+ 9652us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:43:56.259551 (+ 286us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:43:56.259565 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:43:56.259581 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:43:56.259616 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:43:56.266096 (+ 6480us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:43:56.267117 (+ 1021us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:43:56.267126 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:43:56.267129 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:43:56.267286 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:43:56.267674 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:43:56.267680 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:43:56.267684 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:43:56.268020 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:43:56.268343 (+ 323us) server_negotiation.cc:1035] Waiting for connection context
0619 16:43:56.268652 (+ 309us) server_negotiation.cc:299] Negotiation successful
0619 16:43:56.268909 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15671,"thread_start_us":16433,"threads_started":1}
I20260619 16:43:56.270493 7247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:56.270000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:35535 (local address 127.0.0.1:55098)
0619 16:43:56.270215 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0619 16:43:56.270342 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104}
I20260619 16:43:56.270682 7134 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 85d1444e474f4e189618fb22f5a42908 [term 1 LEARNER]: Refusing update from remote peer 5ba5d12f1093420e96661d1c478add9b: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260619 16:43:56.270808 6711 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:43:56.271358 7273 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 5ba5d12f1093420e96661d1c478add9b [LEADER]: Connected to new peer: Peer: permanent_uuid: "85d1444e474f4e189618fb22f5a42908" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42337 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260619 16:43:56.271512 7247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:56.271105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:35535 (local address 127.0.0.1:55114)
0619 16:43:56.271295 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0619 16:43:56.271370 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
W20260619 16:43:56.272403 6514 consensus_peers.cc:487] T 00000000000000000000000000000000 P 5ba5d12f1093420e96661d1c478add9b -> Peer 2e547db70ef542a88009307c8181f39d (127.0.97.60:35535): Couldn't send request to peer 2e547db70ef542a88009307c8181f39d. Status: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35535: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260619 16:43:56.274209 7246 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 5ba5d12f1093420e96661d1c478add9b [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 85d1444e474f4e189618fb22f5a42908 (127.0.97.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "5ba5d12f1093420e96661d1c478add9b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46817 } } peers { permanent_uuid: "85d1444e474f4e189618fb22f5a42908" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42337 } attrs { promote: false } } peers { permanent_uuid: "2e547db70ef542a88009307c8181f39d" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 35535 } attrs { promote: true } } }
I20260619 16:43:56.274860 7133 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 85d1444e474f4e189618fb22f5a42908 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 85d1444e474f4e189618fb22f5a42908 (127.0.97.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "5ba5d12f1093420e96661d1c478add9b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46817 } } peers { permanent_uuid: "85d1444e474f4e189618fb22f5a42908" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42337 } attrs { promote: false } } peers { permanent_uuid: "2e547db70ef542a88009307c8181f39d" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 35535 } attrs { promote: true } } }
I20260619 16:43:56.277688 7246 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5ba5d12f1093420e96661d1c478add9b [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "5ba5d12f1093420e96661d1c478add9b" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "5ba5d12f1093420e96661d1c478add9b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46817 } } peers { permanent_uuid: "85d1444e474f4e189618fb22f5a42908" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42337 } attrs { promote: false } } peers { permanent_uuid: "2e547db70ef542a88009307c8181f39d" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 35535 } attrs { promote: true } } }
I20260619 16:43:56.278149 7246 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5ba5d12f1093420e96661d1c478add9b [sys.catalog]: This master's current role is: LEADER
I20260619 16:43:56.277911 7274 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5ba5d12f1093420e96661d1c478add9b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5ba5d12f1093420e96661d1c478add9b" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "5ba5d12f1093420e96661d1c478add9b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46817 } } peers { permanent_uuid: "85d1444e474f4e189618fb22f5a42908" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42337 } attrs { promote: false } } peers { permanent_uuid: "2e547db70ef542a88009307c8181f39d" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 35535 } attrs { promote: true } } }
I20260619 16:43:56.278275 7274 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5ba5d12f1093420e96661d1c478add9b [sys.catalog]: This master's current role is: LEADER
I20260619 16:43:56.278357 7162 sys_catalog.cc:455] T 00000000000000000000000000000000 P 85d1444e474f4e189618fb22f5a42908 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5ba5d12f1093420e96661d1c478add9b" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "5ba5d12f1093420e96661d1c478add9b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46817 } } peers { permanent_uuid: "85d1444e474f4e189618fb22f5a42908" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42337 } attrs { promote: false } } peers { permanent_uuid: "2e547db70ef542a88009307c8181f39d" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 35535 } attrs { promote: true } } }
I20260619 16:43:56.278587 7162 sys_catalog.cc:458] T 00000000000000000000000000000000 P 85d1444e474f4e189618fb22f5a42908 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 16:43:56.279498 7087 fs_manager.cc:709] Time spent opening directory manager: real 0.017s user 0.000s sys 0.008s
I20260619 16:43:56.281265 7280 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:43:56.281651 7087 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887385903558-388-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:43:56.281939 7087 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.001s
I20260619 16:43:56.282043 7087 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887385903558-388-0/minicluster-data/master-2/data,/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887385903558-388-0/minicluster-data/master-2/wal
uuid: "2e547db70ef542a88009307c8181f39d"
format_stamp: "Formatted at 2026-06-19 16:43:56 on dist-test-slave-wvz7"
server_key: "569c9749205ec3d458c9d0ceeaa154b8"
server_key_iv: "9d2081e8d2c49a5feb94ac45a3370f39"
server_key_version: "encryptionkey@0"
I20260619 16:43:56.288095 7087 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 2e547db70ef542a88009307c8181f39d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:43:56.294627 7087 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 2e547db70ef542a88009307c8181f39d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260619 16:43:56.294790 7087 log.cc:1196] T 00000000000000000000000000000000 P 2e547db70ef542a88009307c8181f39d: Deleting WAL directory at /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887385903558-388-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260619 16:43:56.295223 7087 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 2e547db70ef542a88009307c8181f39d: Deleting consensus metadata
I20260619 16:43:56.295455 7087 master_runner.cc:315] Copying system tablet from 127.0.97.62:46817
I20260619 16:43:56.297978 7087 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 2e547db70ef542a88009307c8181f39d: tablet copy: Beginning tablet copy session from remote peer at address 127.0.97.62:46817
I20260619 16:43:56.312865 6718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:56.298759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46088 (local address 127.0.97.62:46817)
0619 16:43:56.298975 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0619 16:43:56.298981 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:43:56.299441 (+ 460us) server_negotiation.cc:407] Connection header received
0619 16:43:56.299590 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:43:56.299596 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:43:56.299673 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:43:56.299778 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:43:56.300718 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:56.302319 (+ 1601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:56.304097 (+ 1778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:56.304728 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:56.304858 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:43:56.306971 (+ 2113us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:43:56.306977 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:43:56.306982 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:43:56.307018 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:43:56.310205 (+ 3187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:43:56.310937 (+ 732us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:43:56.310942 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:43:56.310947 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:43:56.311086 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:43:56.311456 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:43:56.311460 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:43:56.311464 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:43:56.311753 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:43:56.311892 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0619 16:43:56.312316 (+ 424us) server_negotiation.cc:299] Negotiation successful
0619 16:43:56.312523 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97}
I20260619 16:43:56.312928 7286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:56.298736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46817 (local address 127.0.0.1:46088)
0619 16:43:56.299314 (+ 578us) negotiation.cc:107] Waiting for socket to connect
0619 16:43:56.299342 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0619 16:43:56.299481 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:43:56.299813 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:43:56.299818 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:43:56.300303 (+ 485us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:43:56.300564 (+ 261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:43:56.300576 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:43:56.302472 (+ 1896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:43:56.302478 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:43:56.303952 (+ 1474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:43:56.303966 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:43:56.304891 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:43:56.304897 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:43:56.305074 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:43:56.305757 (+ 683us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:43:56.305783 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:43:56.306777 (+ 994us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:43:56.310339 (+ 3562us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:43:56.310345 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:43:56.310349 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:43:56.310819 (+ 470us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:43:56.311189 (+ 370us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:43:56.311194 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:43:56.311197 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:43:56.311361 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:43:56.311870 (+ 509us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:43:56.311875 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:43:56.312049 (+ 174us) client_negotiation.cc:769] Sending connection context
0619 16:43:56.312194 (+ 145us) client_negotiation.cc:240] Negotiation successful
0619 16:43:56.312483 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":460,"thread_start_us":338,"threads_started":1}
I20260619 16:43:56.313910 6547 tablet_copy_service.cc:140] P 5ba5d12f1093420e96661d1c478add9b: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 2e547db70ef542a88009307c8181f39d ({username='oryx', principal='oryx/127.0.97.60@KRBTEST.COM'} at 127.0.0.1:46088)
I20260619 16:43:56.314071 6547 tablet_copy_service.cc:161] P 5ba5d12f1093420e96661d1c478add9b: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 2e547db70ef542a88009307c8181f39d at {username='oryx', principal='oryx/127.0.97.60@KRBTEST.COM'} at 127.0.0.1:46088: session id = 2e547db70ef542a88009307c8181f39d-00000000000000000000000000000000
I20260619 16:43:56.317641 6547 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 5ba5d12f1093420e96661d1c478add9b: Tablet Copy: opened 0 blocks and 1 log segments
I20260619 16:43:56.319458 7087 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260619 16:43:56.325946 7087 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 2e547db70ef542a88009307c8181f39d: tablet copy: Starting download of 0 data blocks...
I20260619 16:43:56.326231 7087 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 2e547db70ef542a88009307c8181f39d: tablet copy: Starting download of 1 WAL segments...
I20260619 16:43:56.328694 7087 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 2e547db70ef542a88009307c8181f39d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260619 16:43:56.333173 6547 tablet_copy_service.cc:342] P 5ba5d12f1093420e96661d1c478add9b: Request end of tablet copy session 2e547db70ef542a88009307c8181f39d-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.97.60@KRBTEST.COM'} at 127.0.0.1:46088
I20260619 16:43:56.333323 6547 tablet_copy_service.cc:433] P 5ba5d12f1093420e96661d1c478add9b: ending tablet copy session 2e547db70ef542a88009307c8181f39d-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 2e547db70ef542a88009307c8181f39d
I20260619 16:43:56.346283 7087 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:43:56.348977 7287 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:43:56.358982 7288 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:43:56.359331 7087 server_base.cc:1034] running on GCE node
W20260619 16:43:56.359889 7290 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:43:56.360306 7087 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:43:56.361043 7087 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:43:56.362352 7087 hybrid_clock.cc:648] HybridClock initialized: now 1781887436362302 us; error 38 us; skew 500 ppm
I20260619 16:43:56.362660 7087 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:43:56 dist-test-slave-wvz7 krb5kdc[6482](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887436, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:43:56.364845 7087 init.cc:377] Logged in from keytab as oryx/127.0.97.60@KRBTEST.COM (short username oryx)
I20260619 16:43:56.366091 7087 webserver.cc:460] Webserver started at http://127.0.97.60:34953/ using document root <none> and password file <none>
I20260619 16:43:56.366386 7087 fs_manager.cc:359] Metadata directory not provided
I20260619 16:43:56.366479 7087 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:43:56.369443 7087 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260619 16:43:56.371694 7297 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:43:56.372210 7087 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260619 16:43:56.372330 7087 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887385903558-388-0/minicluster-data/master-2/data,/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887385903558-388-0/minicluster-data/master-2/wal
uuid: "2e547db70ef542a88009307c8181f39d"
format_stamp: "Formatted at 2026-06-19 16:43:56 on dist-test-slave-wvz7"
server_key: "569c9749205ec3d458c9d0ceeaa154b8"
server_key_iv: "9d2081e8d2c49a5feb94ac45a3370f39"
server_key_version: "encryptionkey@0"
I20260619 16:43:56.372505 7087 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887385903558-388-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887385903558-388-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887385903558-388-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:43:56.388455 7087 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:43:56.390376 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu with pid 6588
I20260619 16:43:56.392215 7087 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:43:56.410177 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu
/tmp/dist-test-taskEs2EH8/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887385903558-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887385903558-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887385903558-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887385903558-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:40321
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:46817
--webserver_port=45013
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887385903558-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44537
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.97.62:46817,127.0.97.61:42337
--tserver_master_addrs=127.0.97.62:46817,127.0.97.61:42337,127.0.97.60:35535 with env {KRB5CCNAME=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887385903558-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887385903558-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887385903558-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 16:43:56.423769 7087 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.60:35535
I20260619 16:43:56.423830 7349 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.60:35535 every 8 connection(s)
I20260619 16:43:56.424719 7087 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887385903558-388-0/minicluster-data/master-2/data/info.pb
I20260619 16:43:56.430255 7350 sys_catalog.cc:263] Verifying existing consensus state
I20260619 16:43:56.434296 7350 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2e547db70ef542a88009307c8181f39d: Bootstrap starting.
I20260619 16:43:56.451828 7350 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2e547db70ef542a88009307c8181f39d: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 16:43:56.452279 7350 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2e547db70ef542a88009307c8181f39d: Bootstrap complete.
I20260619 16:43:56.453256 7350 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2e547db70ef542a88009307c8181f39d [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "5ba5d12f1093420e96661d1c478add9b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46817 } } peers { permanent_uuid: "85d1444e474f4e189618fb22f5a42908" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42337 } attrs { promote: false } } peers { permanent_uuid: "2e547db70ef542a88009307c8181f39d" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 35535 } attrs { promote: true } }
I20260619 16:43:56.453552 7350 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2e547db70ef542a88009307c8181f39d [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 2e547db70ef542a88009307c8181f39d, State: Initialized, Role: LEARNER
I20260619 16:43:56.453842 7350 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2e547db70ef542a88009307c8181f39d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "5ba5d12f1093420e96661d1c478add9b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46817 } } peers { permanent_uuid: "85d1444e474f4e189618fb22f5a42908" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42337 } attrs { promote: false } } peers { permanent_uuid: "2e547db70ef542a88009307c8181f39d" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 35535 } attrs { promote: true } }
I20260619 16:43:56.454648 7353 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2e547db70ef542a88009307c8181f39d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "5ba5d12f1093420e96661d1c478add9b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46817 } } peers { permanent_uuid: "85d1444e474f4e189618fb22f5a42908" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42337 } attrs { promote: false } } peers { permanent_uuid: "2e547db70ef542a88009307c8181f39d" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 35535 } attrs { promote: true } } }
I20260619 16:43:56.454883 7350 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2e547db70ef542a88009307c8181f39d [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:43:56.455004 7353 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2e547db70ef542a88009307c8181f39d [sys.catalog]: This master's current role is: LEARNER
I20260619 16:43:56.462916 7364 catalog_manager.cc:1259] Loaded cluster ID: dc389b63eb214c9bb142cff4aef89cd5
I20260619 16:43:56.463016 7364 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 2e547db70ef542a88009307c8181f39d: loading cluster ID for follower catalog manager: success
I20260619 16:43:56.466153 7364 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 2e547db70ef542a88009307c8181f39d: acquiring CA information for follower catalog manager: success
I20260619 16:43:56.468376 7364 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 2e547db70ef542a88009307c8181f39d: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260619 16:43:56.737901 7319 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:43:56.738173 7319 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:43:56.738306 7319 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:43:56.745817 7319 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:43:56.745930 7319 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:43:56.746089 7319 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260619 16:43:56.758778 7319 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44537
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887385903558-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887385903558-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887385903558-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:40321
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887385903558-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=45013
--tserver_master_addrs=127.0.97.62:46817,127.0.97.61:42337,127.0.97.60:35535
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887385903558-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:43:56.760296 7319 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:43:56.761894 7319 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:43:56.776530 7369 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:43:56.776554 7370 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:43:56.776715 7372 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:43:56.777464 7319 server_base.cc:1034] running on GCE node
I20260619 16:43:56.778076 7319 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:43:56.779238 7319 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:43:56.780474 7319 hybrid_clock.cc:648] HybridClock initialized: now 1781887436780410 us; error 46 us; skew 500 ppm
I20260619 16:43:56.780812 7319 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:43:56 dist-test-slave-wvz7 krb5kdc[6482](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887436, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:43:56.786947 7319 init.cc:377] Logged in from keytab as oryx/127.0.97.1@KRBTEST.COM (short username oryx)
I20260619 16:43:56.788421 7319 webserver.cc:460] Webserver started at http://127.0.97.1:45013/ using document root <none> and password file <none>
I20260619 16:43:56.789076 7319 fs_manager.cc:359] Metadata directory not provided
I20260619 16:43:56.789187 7319 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:43:56.795024 7319 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20260619 16:43:56.797793 7380 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:43:56.799137 7319 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260619 16:43:56.799276 7319 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887385903558-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887385903558-388-0/minicluster-data/ts-0/wal
uuid: "4e87af70e6654dff8d6f88b6b739ee42"
format_stamp: "Formatted at 2026-06-19 16:43:53 on dist-test-slave-wvz7"
server_key: "f4ca3b4fc3c1cbfc574a36fabd0073fb"
server_key_iv: "bcbd86ec675b13518f137947b870222c"
server_key_version: "encryptionkey@0"
I20260619 16:43:56.800192 7319 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887385903558-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887385903558-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887385903558-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:43:56.823814 7319 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:43:56.827478 7319 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:43:56.827713 7319 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:43:56.828634 7319 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:43:56.830307 7319 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:43:56.830384 7319 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:43:56.830475 7319 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:43:56.830545 7319 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:43:56.861552 7319 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:40321
I20260619 16:43:56.861616 7492 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:40321 every 8 connection(s)
I20260619 16:43:56.862898 7319 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887385903558-388-0/minicluster-data/ts-0/data/info.pb
I20260619 16:43:56.871497 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu as pid 7319
I20260619 16:43:56.871855 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu pid 6722
Jun 19 16:43:56 dist-test-slave-wvz7 krb5kdc[6482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887436, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.60@KRBTEST.COM
Jun 19 16:43:56 dist-test-slave-wvz7 krb5kdc[6482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887436, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
I20260619 16:43:56.890079 7497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:56.865725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:38389 (local address 127.0.97.60:35535)
0619 16:43:56.866664 (+ 939us) server_negotiation.cc:206] Beginning negotiation
0619 16:43:56.866669 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:43:56.870186 (+ 3517us) server_negotiation.cc:407] Connection header received
0619 16:43:56.870411 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:43:56.870416 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:43:56.870488 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:43:56.870561 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:43:56.872760 (+ 2199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:56.874331 (+ 1571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:56.876974 (+ 2643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:56.877659 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:56.877770 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:43:56.882564 (+ 4794us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:43:56.882572 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:43:56.882576 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:43:56.882615 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:43:56.886377 (+ 3762us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:43:56.887344 (+ 967us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:43:56.887350 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:43:56.887353 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:43:56.887486 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:43:56.888064 (+ 578us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:43:56.888069 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:43:56.888072 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:43:56.888428 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:43:56.888540 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0619 16:43:56.889504 (+ 964us) server_negotiation.cc:299] Negotiation successful
0619 16:43:56.889710 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":803,"thread_start_us":669,"threads_started":1}
I20260619 16:43:56.892495 7499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:56.866295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:35535 (local address 127.0.97.1:38389)
0619 16:43:56.870101 (+ 3806us) negotiation.cc:107] Waiting for socket to connect
0619 16:43:56.870117 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 16:43:56.870284 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:43:56.870808 (+ 524us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:43:56.870812 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:43:56.871663 (+ 851us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:43:56.872569 (+ 906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:43:56.872597 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:43:56.874556 (+ 1959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:43:56.874561 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:43:56.876798 (+ 2237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:43:56.876816 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:43:56.878176 (+ 1360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:43:56.878183 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:43:56.878493 (+ 310us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:43:56.879220 (+ 727us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:43:56.879250 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:43:56.882325 (+ 3075us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:43:56.886529 (+ 4204us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:43:56.886538 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:43:56.886556 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:43:56.887188 (+ 632us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:43:56.887630 (+ 442us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:43:56.887634 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:43:56.887638 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:43:56.887870 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:43:56.888644 (+ 774us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:43:56.888665 (+ 21us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:43:56.889060 (+ 395us) client_negotiation.cc:769] Sending connection context
0619 16:43:56.890600 (+ 1540us) client_negotiation.cc:240] Negotiation successful
0619 16:43:56.891091 (+ 491us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3636,"mutex_wait_us":313,"thread_start_us":758,"threads_started":1}
I20260619 16:43:56.894690 7498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:56.865712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:38781 (local address 127.0.97.61:42337)
0619 16:43:56.866908 (+ 1196us) server_negotiation.cc:206] Beginning negotiation
0619 16:43:56.866912 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:43:56.867851 (+ 939us) server_negotiation.cc:407] Connection header received
0619 16:43:56.869652 (+ 1801us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:43:56.869657 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:43:56.869794 (+ 137us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:43:56.869898 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:43:56.880447 (+ 10549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:56.881254 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:56.884100 (+ 2846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:56.885007 (+ 907us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:56.885130 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:43:56.888752 (+ 3622us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:43:56.888758 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:43:56.888762 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:43:56.888800 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:43:56.891893 (+ 3093us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:43:56.892623 (+ 730us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:43:56.892629 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:43:56.892633 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:43:56.892773 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:43:56.893181 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:43:56.893185 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:43:56.893188 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:43:56.893502 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:43:56.893598 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0619 16:43:56.894145 (+ 547us) server_negotiation.cc:299] Negotiation successful
0619 16:43:56.894348 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1109,"thread_start_us":997,"threads_started":1}
I20260619 16:43:56.895514 7500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:56.866295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:42337 (local address 127.0.97.1:38781)
0619 16:43:56.867697 (+ 1402us) negotiation.cc:107] Waiting for socket to connect
0619 16:43:56.867737 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0619 16:43:56.869400 (+ 1663us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:43:56.870343 (+ 943us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:43:56.870355 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:43:56.879373 (+ 9018us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:43:56.879658 (+ 285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:43:56.879673 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:43:56.881475 (+ 1802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:43:56.881481 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:43:56.882553 (+ 1072us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:43:56.882566 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:43:56.885383 (+ 2817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:43:56.885389 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:43:56.885558 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:43:56.886124 (+ 566us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:43:56.886147 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:43:56.888541 (+ 2394us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:43:56.892060 (+ 3519us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:43:56.892065 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:43:56.892068 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:43:56.892489 (+ 421us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:43:56.892901 (+ 412us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:43:56.892905 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:43:56.892908 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:43:56.893065 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:43:56.893670 (+ 605us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:43:56.893674 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:43:56.893895 (+ 221us) client_negotiation.cc:769] Sending connection context
0619 16:43:56.894874 (+ 979us) client_negotiation.cc:240] Negotiation successful
0619 16:43:56.895140 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1180,"thread_start_us":3390,"threads_started":1}
I20260619 16:43:56.897330 7495 heartbeater.cc:347] Connected to a master server at 127.0.97.61:42337
I20260619 16:43:56.897653 7495 heartbeater.cc:464] Registering TS with master...
I20260619 16:43:56.898615 7495 heartbeater.cc:511] Master 127.0.97.61:42337 requested a full tablet report, sending...
Jun 19 16:43:56 dist-test-slave-wvz7 krb5kdc[6482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887433, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.2@KRBTEST.COM
I20260619 16:43:56.901021 7124 ts_manager.cc:194] Registered new tserver with Master: 4e87af70e6654dff8d6f88b6b739ee42 (127.0.97.1:40321)
I20260619 16:43:56.902565 7493 heartbeater.cc:347] Connected to a master server at 127.0.97.60:35535
I20260619 16:43:56.902688 7493 heartbeater.cc:464] Registering TS with master...
I20260619 16:43:56.903074 7493 heartbeater.cc:511] Master 127.0.97.60:35535 requested a full tablet report, sending...
I20260619 16:43:56.904904 7314 ts_manager.cc:194] Registered new tserver with Master: 4e87af70e6654dff8d6f88b6b739ee42 (127.0.97.1:40321)
Jun 19 16:43:56 dist-test-slave-wvz7 krb5kdc[6482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887436, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260619 16:43:56.907862 7497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:56.887131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55122 (local address 127.0.97.60:35535)
0619 16:43:56.890538 (+ 3407us) server_negotiation.cc:206] Beginning negotiation
0619 16:43:56.890545 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:43:56.894105 (+ 3560us) server_negotiation.cc:407] Connection header received
0619 16:43:56.894708 (+ 603us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:43:56.894714 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:43:56.894813 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:43:56.894891 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:43:56.902518 (+ 7627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:56.903502 (+ 984us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:56.905847 (+ 2345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:56.906670 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:56.906800 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:43:56.906990 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0619 16:43:56.907453 (+ 463us) server_negotiation.cc:299] Negotiation successful
0619 16:43:56.907593 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3302,"thread_start_us":14645,"threads_started":1}
I20260619 16:43:56.908174 7505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:56.886810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:35535 (local address 127.0.0.1:55122)
0619 16:43:56.894011 (+ 7201us) negotiation.cc:107] Waiting for socket to connect
0619 16:43:56.894030 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 16:43:56.894214 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:43:56.895113 (+ 899us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:43:56.895118 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:43:56.895147 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:43:56.896327 (+ 1180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:43:56.896346 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:43:56.903678 (+ 7332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:43:56.903683 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:43:56.905706 (+ 2023us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:43:56.905718 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:43:56.907091 (+ 1373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:43:56.907097 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:43:56.907274 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:43:56.907308 (+ 34us) client_negotiation.cc:769] Sending connection context
0619 16:43:56.907387 (+ 79us) client_negotiation.cc:240] Negotiation successful
0619 16:43:56.907577 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7094,"thread_start_us":7011,"threads_started":1}
I20260619 16:43:56.909214 7504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:56.872865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46542 (local address 127.0.97.2:34331)
0619 16:43:56.889245 (+ 16380us) server_negotiation.cc:206] Beginning negotiation
0619 16:43:56.889255 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 16:43:56.889283 (+ 28us) server_negotiation.cc:407] Connection header received
0619 16:43:56.889347 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:43:56.889356 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:43:56.889618 (+ 262us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:43:56.889803 (+ 185us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:43:56.892055 (+ 2252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:56.893577 (+ 1522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:56.895829 (+ 2252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:56.896471 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:56.896584 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:43:56.901866 (+ 5282us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:43:56.901880 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:43:56.901896 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:43:56.901939 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:43:56.905790 (+ 3851us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:43:56.906605 (+ 815us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:43:56.906614 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:43:56.906619 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:43:56.906864 (+ 245us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:43:56.907314 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:43:56.907319 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:43:56.907322 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:43:56.907776 (+ 454us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:43:56.908044 (+ 268us) server_negotiation.cc:1035] Waiting for connection context
0619 16:43:56.908587 (+ 543us) server_negotiation.cc:299] Negotiation successful
0619 16:43:56.908822 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16202,"thread_start_us":18933,"threads_started":1}
I20260619 16:43:56.909103 7325 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 2e547db70ef542a88009307c8181f39d [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260619 16:43:56.909830 7353 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2e547db70ef542a88009307c8181f39d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5ba5d12f1093420e96661d1c478add9b. Latest consensus state: current_term: 1 leader_uuid: "5ba5d12f1093420e96661d1c478add9b" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "5ba5d12f1093420e96661d1c478add9b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46817 } } peers { permanent_uuid: "85d1444e474f4e189618fb22f5a42908" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42337 } attrs { promote: false } } peers { permanent_uuid: "2e547db70ef542a88009307c8181f39d" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 35535 } attrs { promote: true } } }
I20260619 16:43:56.910068 7353 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2e547db70ef542a88009307c8181f39d [sys.catalog]: This master's current role is: LEARNER
I20260619 16:43:56.910627 6845 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:43:56.914081 7499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:56.868210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46817 (local address 127.0.97.1:36105)
0619 16:43:56.893095 (+ 24885us) negotiation.cc:107] Waiting for socket to connect
0619 16:43:56.893112 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 16:43:56.893187 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:43:56.898011 (+ 4824us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:43:56.898016 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:43:56.898494 (+ 478us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:43:56.898688 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:43:56.898699 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:43:56.900283 (+ 1584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:43:56.900288 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:43:56.901635 (+ 1347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:43:56.901647 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:43:56.902672 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:43:56.902682 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:43:56.903360 (+ 678us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:43:56.905661 (+ 2301us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:43:56.905684 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:43:56.908080 (+ 2396us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:43:56.911495 (+ 3415us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:43:56.911501 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:43:56.911505 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:43:56.912022 (+ 517us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:43:56.912441 (+ 419us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:43:56.912447 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:43:56.912451 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:43:56.912599 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:43:56.913244 (+ 645us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:43:56.913248 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:43:56.913368 (+ 120us) client_negotiation.cc:769] Sending connection context
0619 16:43:56.913469 (+ 101us) client_negotiation.cc:240] Negotiation successful
0619 16:43:56.913662 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24778,"thread_start_us":28831,"threads_started":1}
I20260619 16:43:56.914212 7502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:56.868380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:36105 (local address 127.0.97.62:46817)
0619 16:43:56.897620 (+ 29240us) server_negotiation.cc:206] Beginning negotiation
0619 16:43:56.897626 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:43:56.897643 (+ 17us) server_negotiation.cc:407] Connection header received
0619 16:43:56.897686 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:43:56.897691 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:43:56.897848 (+ 157us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:43:56.898881 (+ 1033us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:43:56.899063 (+ 182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:56.900126 (+ 1063us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:56.901915 (+ 1789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:56.902500 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:56.902973 (+ 473us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:43:56.908249 (+ 5276us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:43:56.908254 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:43:56.908257 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:43:56.908289 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:43:56.911329 (+ 3040us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:43:56.912179 (+ 850us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:43:56.912185 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:43:56.912189 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:43:56.912326 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:43:56.912737 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:43:56.912742 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:43:56.912745 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:43:56.913087 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:43:56.913258 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0619 16:43:56.913584 (+ 326us) server_negotiation.cc:299] Negotiation successful
0619 16:43:56.913834 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29123,"thread_start_us":28979,"threads_started":1}
I20260619 16:43:56.918386 7494 heartbeater.cc:347] Connected to a master server at 127.0.97.62:46817
I20260619 16:43:56.918510 7494 heartbeater.cc:464] Registering TS with master...
I20260619 16:43:56.918915 7494 heartbeater.cc:511] Master 127.0.97.62:46817 requested a full tablet report, sending...
I20260619 16:43:56.920074 6527 ts_manager.cc:194] Re-registered known tserver with Master: 4e87af70e6654dff8d6f88b6b739ee42 (127.0.97.1:40321)
I20260619 16:43:56.921175 6527 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:36105
I20260619 16:43:56.991925 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu with pid 6722
I20260619 16:43:57.004354 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu
/tmp/dist-test-taskEs2EH8/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887385903558-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887385903558-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887385903558-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887385903558-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:34331
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:46817
--webserver_port=37687
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887385903558-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44537
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.97.62:46817,127.0.97.61:42337
--tserver_master_addrs=127.0.97.62:46817,127.0.97.61:42337,127.0.97.60:35535 with env {KRB5CCNAME=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887385903558-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887385903558-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887385903558-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:43:57.311007 7508 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:43:57.311262 7508 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:43:57.311381 7508 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:43:57.318562 7508 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:43:57.318671 7508 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:43:57.318820 7508 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260619 16:43:57.328585 7508 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44537
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887385903558-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887385903558-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887385903558-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:34331
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887385903558-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=37687
--tserver_master_addrs=127.0.97.62:46817,127.0.97.61:42337,127.0.97.60:35535
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887385903558-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:43:57.329860 7508 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:43:57.331103 7508 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:43:57.343619 7513 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:43:57.343608 7514 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:43:57.343636 7516 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:43:57.344640 7508 server_base.cc:1034] running on GCE node
I20260619 16:43:57.345192 7508 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:43:57.346122 7508 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:43:57.347353 7508 hybrid_clock.cc:648] HybridClock initialized: now 1781887437347283 us; error 49 us; skew 500 ppm
I20260619 16:43:57.347720 7508 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:43:57 dist-test-slave-wvz7 krb5kdc[6482](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887437, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:43:57.350796 7508 init.cc:377] Logged in from keytab as oryx/127.0.97.2@KRBTEST.COM (short username oryx)
I20260619 16:43:57.352228 7508 webserver.cc:460] Webserver started at http://127.0.97.2:37687/ using document root <none> and password file <none>
I20260619 16:43:57.352885 7508 fs_manager.cc:359] Metadata directory not provided
I20260619 16:43:57.353008 7508 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:43:57.358335 7508 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260619 16:43:57.360792 7523 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:43:57.362146 7508 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 16:43:57.362296 7508 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887385903558-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887385903558-388-0/minicluster-data/ts-1/wal
uuid: "449f7e56e3754b22a114d8f19ed82f76"
format_stamp: "Formatted at 2026-06-19 16:43:54 on dist-test-slave-wvz7"
server_key: "957a080faef101cc972bc289c8bc2c1e"
server_key_iv: "2e02218e7d0bf01ea06e5f9a5461ee19"
server_key_version: "encryptionkey@0"
I20260619 16:43:57.363209 7508 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887385903558-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887385903558-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887385903558-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:43:57.392606 7508 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:43:57.396557 7508 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:43:57.396807 7508 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:43:57.397706 7508 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:43:57.399020 7508 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:43:57.399112 7508 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:43:57.399211 7508 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:43:57.399277 7508 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:43:57.430356 7508 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:34331
I20260619 16:43:57.430398 7635 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:34331 every 8 connection(s)
I20260619 16:43:57.431891 7508 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887385903558-388-0/minicluster-data/ts-1/data/info.pb
I20260619 16:43:57.432307 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu as pid 7508
I20260619 16:43:57.432562 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu pid 6855
Jun 19 16:43:57 dist-test-slave-wvz7 krb5kdc[6482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887437, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
Jun 19 16:43:57 dist-test-slave-wvz7 krb5kdc[6482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887437, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
Jun 19 16:43:57 dist-test-slave-wvz7 krb5kdc[6482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887437, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.60@KRBTEST.COM
Jun 19 16:43:57 dist-test-slave-wvz7 krb5kdc[6482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887433, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.3@KRBTEST.COM
I20260619 16:43:57.472229 7643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:57.439465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:35357 (local address 127.0.97.62:46817)
0619 16:43:57.452409 (+ 12944us) server_negotiation.cc:206] Beginning negotiation
0619 16:43:57.452415 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:43:57.452438 (+ 23us) server_negotiation.cc:407] Connection header received
0619 16:43:57.452502 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:43:57.452508 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:43:57.452586 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:43:57.452720 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:43:57.455209 (+ 2489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:57.456190 (+ 981us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:57.458954 (+ 2764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:57.459566 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:57.459678 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:43:57.464546 (+ 4868us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:43:57.464552 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:43:57.464557 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:43:57.464598 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:43:57.468227 (+ 3629us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:43:57.469617 (+ 1390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:43:57.469622 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:43:57.469625 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:43:57.469807 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:43:57.470277 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:43:57.470282 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:43:57.470286 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:43:57.470613 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:43:57.470748 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0619 16:43:57.471680 (+ 932us) server_negotiation.cc:299] Negotiation successful
0619 16:43:57.471901 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12817,"thread_start_us":13357,"threads_started":1}
I20260619 16:43:57.472345 7647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:57.447894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:36967 (local address 127.0.97.61:42337)
0619 16:43:57.455167 (+ 7273us) server_negotiation.cc:206] Beginning negotiation
0619 16:43:57.455173 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:43:57.455675 (+ 502us) server_negotiation.cc:407] Connection header received
0619 16:43:57.455854 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:43:57.455860 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:43:57.455939 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:43:57.456037 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:43:57.457395 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:57.458509 (+ 1114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:57.460275 (+ 1766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:57.460799 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:57.460888 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:43:57.465100 (+ 4212us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:43:57.465107 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:43:57.465112 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:43:57.465151 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:43:57.468822 (+ 3671us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:43:57.469710 (+ 888us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:43:57.469715 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:43:57.469719 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:43:57.469891 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:43:57.470328 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:43:57.470333 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:43:57.470336 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:43:57.470675 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:43:57.470792 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0619 16:43:57.471672 (+ 880us) server_negotiation.cc:299] Negotiation successful
0619 16:43:57.471886 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7158,"thread_start_us":7051,"threads_started":1}
I20260619 16:43:57.473470 7640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:57.435538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:35535 (local address 127.0.97.2:39465)
0619 16:43:57.436490 (+ 952us) negotiation.cc:107] Waiting for socket to connect
0619 16:43:57.436557 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0619 16:43:57.439146 (+ 2589us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:43:57.457444 (+ 18298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:43:57.457449 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:43:57.457904 (+ 455us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:43:57.458165 (+ 261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:43:57.458179 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:43:57.459759 (+ 1580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:43:57.459765 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:43:57.461240 (+ 1475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:43:57.461252 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:43:57.462232 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:43:57.462238 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:43:57.462465 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:43:57.462965 (+ 500us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:43:57.462985 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:43:57.465365 (+ 2380us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:43:57.469709 (+ 4344us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:43:57.469713 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:43:57.469717 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:43:57.470153 (+ 436us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:43:57.470540 (+ 387us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:43:57.470545 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:43:57.470548 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:43:57.470690 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:43:57.471256 (+ 566us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:43:57.471259 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:43:57.471396 (+ 137us) client_negotiation.cc:769] Sending connection context
0619 16:43:57.471556 (+ 160us) client_negotiation.cc:240] Negotiation successful
0619 16:43:57.472042 (+ 486us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":592,"spinlock_wait_cycles":9984,"thread_start_us":692,"threads_started":1}
I20260619 16:43:57.473516 7646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:57.447428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:42337 (local address 127.0.97.2:36967)
0619 16:43:57.455564 (+ 8136us) negotiation.cc:107] Waiting for socket to connect
0619 16:43:57.455582 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 16:43:57.455715 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:43:57.456190 (+ 475us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:43:57.456194 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:43:57.456873 (+ 679us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:43:57.457229 (+ 356us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:43:57.457242 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:43:57.458691 (+ 1449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:43:57.458696 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:43:57.460116 (+ 1420us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:43:57.460127 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:43:57.461159 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:43:57.461164 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:43:57.461349 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:43:57.461911 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:43:57.461931 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:43:57.464870 (+ 2939us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:43:57.468965 (+ 4095us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:43:57.468970 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:43:57.468974 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:43:57.469564 (+ 590us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:43:57.470033 (+ 469us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:43:57.470037 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:43:57.470041 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:43:57.470212 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:43:57.470878 (+ 666us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:43:57.470883 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:43:57.471268 (+ 385us) client_negotiation.cc:769] Sending connection context
0619 16:43:57.472522 (+ 1254us) client_negotiation.cc:240] Negotiation successful
0619 16:43:57.472863 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8020,"mutex_wait_us":6,"spinlock_wait_cycles":38016,"thread_start_us":7860,"threads_started":1}
I20260619 16:43:57.473537 7642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:57.439343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46817 (local address 127.0.97.2:35357)
0619 16:43:57.451843 (+ 12500us) negotiation.cc:107] Waiting for socket to connect
0619 16:43:57.451865 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0619 16:43:57.452015 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:43:57.453047 (+ 1032us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:43:57.453061 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:43:57.454074 (+ 1013us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:43:57.454977 (+ 903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:43:57.455003 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:43:57.456362 (+ 1359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:43:57.456367 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:43:57.458780 (+ 2413us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:43:57.458791 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:43:57.459977 (+ 1186us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:43:57.459984 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:43:57.460261 (+ 277us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:43:57.461011 (+ 750us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:43:57.461041 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:43:57.464286 (+ 3245us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:43:57.468373 (+ 4087us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:43:57.468382 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:43:57.468401 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:43:57.469468 (+ 1067us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:43:57.469926 (+ 458us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:43:57.469931 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:43:57.469934 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:43:57.470152 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:43:57.470810 (+ 658us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:43:57.470818 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:43:57.471255 (+ 437us) client_negotiation.cc:769] Sending connection context
0619 16:43:57.472917 (+ 1662us) client_negotiation.cc:240] Negotiation successful
0619 16:43:57.473160 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12331,"mutex_wait_us":27,"thread_start_us":12739,"threads_started":1}
I20260619 16:43:57.475015 7641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:57.436164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:39465 (local address 127.0.97.60:35535)
0619 16:43:57.457126 (+ 20962us) server_negotiation.cc:206] Beginning negotiation
0619 16:43:57.457131 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:43:57.457157 (+ 26us) server_negotiation.cc:407] Connection header received
0619 16:43:57.457208 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:43:57.457213 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:43:57.457286 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:43:57.458331 (+ 1045us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:43:57.458512 (+ 181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:57.459592 (+ 1080us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:57.461424 (+ 1832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:57.462013 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:57.462770 (+ 757us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:43:57.465607 (+ 2837us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:43:57.465614 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:43:57.465618 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:43:57.465656 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:43:57.469575 (+ 3919us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:43:57.470303 (+ 728us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:43:57.470308 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:43:57.470311 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:43:57.470424 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:43:57.470813 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:43:57.470818 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:43:57.470821 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:43:57.471121 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:43:57.474038 (+ 2917us) server_negotiation.cc:1035] Waiting for connection context
0619 16:43:57.474494 (+ 456us) server_negotiation.cc:299] Negotiation successful
0619 16:43:57.474687 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20748,"thread_start_us":20834,"threads_started":1}
I20260619 16:43:57.475106 7639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:57.433687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51218 (local address 127.0.97.3:40501)
0619 16:43:57.454148 (+ 20461us) server_negotiation.cc:206] Beginning negotiation
0619 16:43:57.454156 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 16:43:57.454182 (+ 26us) server_negotiation.cc:407] Connection header received
0619 16:43:57.454233 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:43:57.454239 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:43:57.454447 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:43:57.454579 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:43:57.456802 (+ 2223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:57.458295 (+ 1493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:57.460098 (+ 1803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:57.461257 (+ 1159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:57.461365 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:43:57.467958 (+ 6593us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:43:57.467971 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:43:57.467989 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:43:57.468035 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:43:57.471446 (+ 3411us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:43:57.472240 (+ 794us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:43:57.472248 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:43:57.472251 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:43:57.472389 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:43:57.472843 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:43:57.472847 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:43:57.472850 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:43:57.473227 (+ 377us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:43:57.473471 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0619 16:43:57.474613 (+ 1142us) server_negotiation.cc:299] Negotiation successful
0619 16:43:57.474850 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20310,"thread_start_us":20170,"threads_started":1}
I20260619 16:43:57.475461 7637 heartbeater.cc:347] Connected to a master server at 127.0.97.62:46817
I20260619 16:43:57.475706 7636 heartbeater.cc:347] Connected to a master server at 127.0.97.60:35535
I20260619 16:43:57.475776 7637 heartbeater.cc:464] Registering TS with master...
I20260619 16:43:57.475811 7636 heartbeater.cc:464] Registering TS with master...
I20260619 16:43:57.476366 6978 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:43:57.476372 7644 heartbeater.cc:347] Connected to a master server at 127.0.97.61:42337
I20260619 16:43:57.476495 7644 heartbeater.cc:464] Registering TS with master...
I20260619 16:43:57.477157 7637 heartbeater.cc:511] Master 127.0.97.62:46817 requested a full tablet report, sending...
I20260619 16:43:57.477363 7644 heartbeater.cc:511] Master 127.0.97.61:42337 requested a full tablet report, sending...
I20260619 16:43:57.477522 7636 heartbeater.cc:511] Master 127.0.97.60:35535 requested a full tablet report, sending...
I20260619 16:43:57.478688 6527 ts_manager.cc:194] Re-registered known tserver with Master: 449f7e56e3754b22a114d8f19ed82f76 (127.0.97.2:34331)
I20260619 16:43:57.478811 7124 ts_manager.cc:194] Registered new tserver with Master: 449f7e56e3754b22a114d8f19ed82f76 (127.0.97.2:34331)
I20260619 16:43:57.479002 7314 ts_manager.cc:194] Registered new tserver with Master: 449f7e56e3754b22a114d8f19ed82f76 (127.0.97.2:34331)
I20260619 16:43:57.479743 6527 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:35357
I20260619 16:43:57.498019 7379 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5ba5d12f1093420e96661d1c478add9b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5ba5d12f1093420e96661d1c478add9b" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "5ba5d12f1093420e96661d1c478add9b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46817 } } peers { permanent_uuid: "85d1444e474f4e189618fb22f5a42908" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42337 } attrs { promote: false } } peers { permanent_uuid: "2e547db70ef542a88009307c8181f39d" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 35535 } attrs { promote: true } } }
I20260619 16:43:57.498410 7379 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5ba5d12f1093420e96661d1c478add9b [sys.catalog]: This master's current role is: LEADER
I20260619 16:43:57.499222 7379 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 5ba5d12f1093420e96661d1c478add9b: attempting to promote NON_VOTER 2e547db70ef542a88009307c8181f39d to VOTER
I20260619 16:43:57.499756 7379 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5ba5d12f1093420e96661d1c478add9b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "5ba5d12f1093420e96661d1c478add9b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46817 } } peers { permanent_uuid: "85d1444e474f4e189618fb22f5a42908" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42337 } attrs { promote: false } } peers { permanent_uuid: "2e547db70ef542a88009307c8181f39d" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35535 } attrs { promote: false } }
I20260619 16:43:57.501669 7325 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 2e547db70ef542a88009307c8181f39d [term 1 LEARNER]: Refusing update from remote peer 5ba5d12f1093420e96661d1c478add9b: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260619 16:43:57.501667 7133 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 85d1444e474f4e189618fb22f5a42908 [term 1 FOLLOWER]: Refusing update from remote peer 5ba5d12f1093420e96661d1c478add9b: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260619 16:43:57.502207 7650 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 5ba5d12f1093420e96661d1c478add9b [LEADER]: Connected to new peer: Peer: permanent_uuid: "2e547db70ef542a88009307c8181f39d" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35535 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260619 16:43:57.502622 7649 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 5ba5d12f1093420e96661d1c478add9b [LEADER]: Connected to new peer: Peer: permanent_uuid: "85d1444e474f4e189618fb22f5a42908" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42337 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260619 16:43:57.506322 7649 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 5ba5d12f1093420e96661d1c478add9b [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 2e547db70ef542a88009307c8181f39d (127.0.97.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "5ba5d12f1093420e96661d1c478add9b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46817 } } peers { permanent_uuid: "85d1444e474f4e189618fb22f5a42908" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42337 } attrs { promote: false } } peers { permanent_uuid: "2e547db70ef542a88009307c8181f39d" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35535 } attrs { promote: false } } }
I20260619 16:43:57.507021 7325 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 2e547db70ef542a88009307c8181f39d [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 2e547db70ef542a88009307c8181f39d (127.0.97.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "5ba5d12f1093420e96661d1c478add9b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46817 } } peers { permanent_uuid: "85d1444e474f4e189618fb22f5a42908" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42337 } attrs { promote: false } } peers { permanent_uuid: "2e547db70ef542a88009307c8181f39d" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35535 } attrs { promote: false } } }
I20260619 16:43:57.510568 7649 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5ba5d12f1093420e96661d1c478add9b [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "5ba5d12f1093420e96661d1c478add9b" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "5ba5d12f1093420e96661d1c478add9b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46817 } } peers { permanent_uuid: "85d1444e474f4e189618fb22f5a42908" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42337 } attrs { promote: false } } peers { permanent_uuid: "2e547db70ef542a88009307c8181f39d" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35535 } attrs { promote: false } } }
I20260619 16:43:57.510862 7649 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5ba5d12f1093420e96661d1c478add9b [sys.catalog]: This master's current role is: LEADER
I20260619 16:43:57.511039 7379 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5ba5d12f1093420e96661d1c478add9b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5ba5d12f1093420e96661d1c478add9b" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "5ba5d12f1093420e96661d1c478add9b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46817 } } peers { permanent_uuid: "85d1444e474f4e189618fb22f5a42908" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42337 } attrs { promote: false } } peers { permanent_uuid: "2e547db70ef542a88009307c8181f39d" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35535 } attrs { promote: false } } }
I20260619 16:43:57.511300 7379 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5ba5d12f1093420e96661d1c478add9b [sys.catalog]: This master's current role is: LEADER
I20260619 16:43:57.515235 7379 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5ba5d12f1093420e96661d1c478add9b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5ba5d12f1093420e96661d1c478add9b" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "5ba5d12f1093420e96661d1c478add9b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46817 } } peers { permanent_uuid: "85d1444e474f4e189618fb22f5a42908" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42337 } attrs { promote: false } } peers { permanent_uuid: "2e547db70ef542a88009307c8181f39d" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35535 } attrs { promote: false } } }
I20260619 16:43:57.515451 7379 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5ba5d12f1093420e96661d1c478add9b [sys.catalog]: This master's current role is: LEADER
I20260619 16:43:57.515673 7133 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 85d1444e474f4e189618fb22f5a42908 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 2e547db70ef542a88009307c8181f39d (127.0.97.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "5ba5d12f1093420e96661d1c478add9b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46817 } } peers { permanent_uuid: "85d1444e474f4e189618fb22f5a42908" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42337 } attrs { promote: false } } peers { permanent_uuid: "2e547db70ef542a88009307c8181f39d" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35535 } attrs { promote: false } } }
I20260619 16:43:57.519701 7655 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2e547db70ef542a88009307c8181f39d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5ba5d12f1093420e96661d1c478add9b" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "5ba5d12f1093420e96661d1c478add9b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46817 } } peers { permanent_uuid: "85d1444e474f4e189618fb22f5a42908" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42337 } attrs { promote: false } } peers { permanent_uuid: "2e547db70ef542a88009307c8181f39d" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35535 } attrs { promote: false } } }
I20260619 16:43:57.520115 7655 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2e547db70ef542a88009307c8181f39d [sys.catalog]: This master's current role is: FOLLOWER
I20260619 16:43:57.520370 7656 sys_catalog.cc:455] T 00000000000000000000000000000000 P 85d1444e474f4e189618fb22f5a42908 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5ba5d12f1093420e96661d1c478add9b" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "5ba5d12f1093420e96661d1c478add9b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46817 } } peers { permanent_uuid: "85d1444e474f4e189618fb22f5a42908" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42337 } attrs { promote: false } } peers { permanent_uuid: "2e547db70ef542a88009307c8181f39d" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35535 } attrs { promote: false } } }
I20260619 16:43:57.520713 7656 sys_catalog.cc:458] T 00000000000000000000000000000000 P 85d1444e474f4e189618fb22f5a42908 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 16:43:57.565580 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu with pid 6855
I20260619 16:43:57.578052 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu
/tmp/dist-test-taskEs2EH8/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887385903558-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887385903558-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887385903558-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887385903558-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:40501
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:46817
--webserver_port=37901
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887385903558-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44537
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.97.62:46817,127.0.97.61:42337
--tserver_master_addrs=127.0.97.62:46817,127.0.97.61:42337,127.0.97.60:35535 with env {KRB5CCNAME=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887385903558-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887385903558-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887385903558-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:43:57.879220 7657 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:43:57.879465 7657 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:43:57.879578 7657 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:43:57.886483 7657 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:43:57.886574 7657 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:43:57.886766 7657 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260619 16:43:57.897532 7657 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44537
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887385903558-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887385903558-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887385903558-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:40501
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887385903558-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=37901
--tserver_master_addrs=127.0.97.62:46817,127.0.97.61:42337,127.0.97.60:35535
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887385903558-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:43:57.898722 7657 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:43:57.899919 7657 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:43:57.912169 7662 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:43:57.912221 7665 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:43:57.912252 7663 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:43:57.912675 7657 server_base.cc:1034] running on GCE node
I20260619 16:43:57.913236 7657 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:43:57.914080 7657 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:43:57.915258 7657 hybrid_clock.cc:648] HybridClock initialized: now 1781887437915211 us; error 41 us; skew 500 ppm
I20260619 16:43:57.915511 7657 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:43:57 dist-test-slave-wvz7 krb5kdc[6482](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887437, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:43:57.918292 7657 init.cc:377] Logged in from keytab as oryx/127.0.97.3@KRBTEST.COM (short username oryx)
I20260619 16:43:57.919448 7657 webserver.cc:460] Webserver started at http://127.0.97.3:37901/ using document root <none> and password file <none>
I20260619 16:43:57.920022 7657 fs_manager.cc:359] Metadata directory not provided
I20260619 16:43:57.920120 7657 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:43:57.923388 7494 heartbeater.cc:503] Master 127.0.97.62:46817 was elected leader, sending a full tablet report...
I20260619 16:43:57.925073 7657 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.001s
I20260619 16:43:57.927502 7672 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:43:57.928695 7657 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 16:43:57.928835 7657 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887385903558-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887385903558-388-0/minicluster-data/ts-2/wal
uuid: "399a36acb0b447e8b9c0d6992bc86ea2"
format_stamp: "Formatted at 2026-06-19 16:43:54 on dist-test-slave-wvz7"
server_key: "afbfae96eaddf174673547acc725f829"
server_key_iv: "7d5232cc81be8d93525f98fdc48b6ee9"
server_key_version: "encryptionkey@0"
I20260619 16:43:57.929761 7657 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887385903558-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887385903558-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887385903558-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:43:57.945107 7657 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:43:57.948509 7657 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:43:57.948738 7657 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:43:57.949548 7657 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:43:57.950802 7657 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:43:57.950891 7657 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:43:57.950985 7657 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:43:57.951052 7657 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:43:57.979476 7657 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:40501
I20260619 16:43:57.979539 7784 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:40501 every 8 connection(s)
I20260619 16:43:57.980816 7657 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887385903558-388-0/minicluster-data/ts-2/data/info.pb
I20260619 16:43:57.984685 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu as pid 7657
Jun 19 16:43:57 dist-test-slave-wvz7 krb5kdc[6482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887437, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.60@KRBTEST.COM
Jun 19 16:43:58 dist-test-slave-wvz7 krb5kdc[6482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887437, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
Jun 19 16:43:58 dist-test-slave-wvz7 krb5kdc[6482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887437, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
I20260619 16:43:58.026405 7789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:57.983512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:55123 (local address 127.0.97.60:35535)
0619 16:43:57.984173 (+ 661us) server_negotiation.cc:206] Beginning negotiation
0619 16:43:57.984180 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:43:57.991248 (+ 7068us) server_negotiation.cc:407] Connection header received
0619 16:43:57.994186 (+ 2938us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:43:57.994193 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:43:57.994310 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:43:57.994410 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:43:57.997300 (+ 2890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:57.998496 (+ 1196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:58.000886 (+ 2390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:58.001550 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:58.001688 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:43:58.007436 (+ 5748us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:43:58.007442 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:43:58.007447 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:43:58.007499 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:43:58.010979 (+ 3480us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:43:58.021548 (+ 10569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:43:58.021554 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:43:58.021558 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:43:58.021754 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:43:58.022938 (+ 1184us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:43:58.022944 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:43:58.022948 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:43:58.023355 (+ 407us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:43:58.023480 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0619 16:43:58.025721 (+ 2241us) server_negotiation.cc:299] Negotiation successful
0619 16:43:58.026012 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":533,"thread_start_us":440,"threads_started":1}
I20260619 16:43:58.027969 7790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:57.983756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:48347 (local address 127.0.97.61:42337)
0619 16:43:57.990640 (+ 6884us) server_negotiation.cc:206] Beginning negotiation
0619 16:43:57.990647 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:43:57.998464 (+ 7817us) server_negotiation.cc:407] Connection header received
0619 16:43:57.999415 (+ 951us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:43:57.999421 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:43:57.999497 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:43:57.999566 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:43:58.000756 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:58.001818 (+ 1062us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:58.004415 (+ 2597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:58.004992 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:58.005102 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:43:58.019304 (+ 14202us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:43:58.019312 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:43:58.019318 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:43:58.019378 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:43:58.023821 (+ 4443us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:43:58.024637 (+ 816us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:43:58.024643 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:43:58.024647 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:43:58.024810 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:43:58.025980 (+ 1170us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:43:58.025986 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:43:58.025989 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:43:58.026325 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:43:58.026456 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0619 16:43:58.027353 (+ 897us) server_negotiation.cc:299] Negotiation successful
0619 16:43:58.027591 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6761,"thread_start_us":6617,"threads_started":1}
I20260619 16:43:58.030073 7795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:57.991866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:35909 (local address 127.0.97.62:46817)
0619 16:43:58.001151 (+ 9285us) server_negotiation.cc:206] Beginning negotiation
0619 16:43:58.001157 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:43:58.001175 (+ 18us) server_negotiation.cc:407] Connection header received
0619 16:43:58.001226 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:43:58.001231 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:43:58.001319 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:43:58.001414 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:43:58.002679 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:58.003733 (+ 1054us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:58.006407 (+ 2674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:58.007016 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:58.007134 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:43:58.014518 (+ 7384us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:43:58.014525 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:43:58.014531 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:43:58.014587 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:43:58.019628 (+ 5041us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:43:58.025158 (+ 5530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:43:58.025163 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:43:58.025167 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:43:58.025339 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:43:58.028603 (+ 3264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:43:58.028609 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:43:58.028613 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:43:58.029023 (+ 410us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:43:58.029187 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0619 16:43:58.029497 (+ 310us) server_negotiation.cc:299] Negotiation successful
0619 16:43:58.029704 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9177,"thread_start_us":9093,"threads_started":1}
I20260619 16:43:58.030695 7794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:57.991779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46817 (local address 127.0.97.3:35909)
0619 16:43:57.999061 (+ 7282us) negotiation.cc:107] Waiting for socket to connect
0619 16:43:57.999080 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 16:43:57.999177 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:43:58.001883 (+ 2706us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:43:58.001888 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:43:58.002299 (+ 411us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:43:58.002541 (+ 242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:43:58.002549 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:43:58.004449 (+ 1900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:43:58.004453 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:43:58.006284 (+ 1831us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:43:58.006298 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:43:58.007172 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:43:58.007178 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:43:58.007379 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:43:58.007980 (+ 601us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:43:58.008001 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:43:58.010682 (+ 2681us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:43:58.019773 (+ 9091us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:43:58.019783 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:43:58.019805 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:43:58.020518 (+ 713us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:43:58.025426 (+ 4908us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:43:58.025430 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:43:58.025434 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:43:58.025697 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:43:58.029140 (+ 3443us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:43:58.029145 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:43:58.029252 (+ 107us) client_negotiation.cc:769] Sending connection context
0619 16:43:58.029362 (+ 110us) client_negotiation.cc:240] Negotiation successful
0619 16:43:58.029607 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7167,"thread_start_us":7040,"threads_started":1}
I20260619 16:43:58.030733 7791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:57.984104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:35535 (local address 127.0.97.3:55123)
0619 16:43:57.991050 (+ 6946us) negotiation.cc:107] Waiting for socket to connect
0619 16:43:57.991110 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0619 16:43:57.993842 (+ 2732us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:43:57.994965 (+ 1123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:43:57.994977 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:43:57.996093 (+ 1116us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:43:57.996980 (+ 887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:43:57.997011 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:43:57.998678 (+ 1667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:43:57.998683 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:43:58.000756 (+ 2073us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:43:58.000767 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:43:58.002014 (+ 1247us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:43:58.002022 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:43:58.002343 (+ 321us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:43:58.003130 (+ 787us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:43:58.003222 (+ 92us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:43:58.007196 (+ 3974us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:43:58.019826 (+ 12630us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:43:58.019832 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:43:58.019836 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:43:58.020898 (+ 1062us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:43:58.021935 (+ 1037us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:43:58.021939 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:43:58.021943 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:43:58.022185 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:43:58.023595 (+ 1410us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:43:58.023604 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:43:58.024075 (+ 471us) client_negotiation.cc:769] Sending connection context
0619 16:43:58.026953 (+ 2878us) client_negotiation.cc:240] Negotiation successful
0619 16:43:58.027568 (+ 615us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6620,"mutex_wait_us":3,"thread_start_us":6394,"threads_started":1}
I20260619 16:43:58.031418 7792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:57.984105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:42337 (local address 127.0.97.3:48347)
0619 16:43:57.998373 (+ 14268us) negotiation.cc:107] Waiting for socket to connect
0619 16:43:57.998389 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 16:43:57.999292 (+ 903us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:43:57.999786 (+ 494us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:43:57.999791 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:43:58.000342 (+ 551us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:43:58.000596 (+ 254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:43:58.000607 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:43:58.003007 (+ 2400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:43:58.003018 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:43:58.004297 (+ 1279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:43:58.004311 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:43:58.005332 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:43:58.005339 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:43:58.005490 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:43:58.006054 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:43:58.006075 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:43:58.018329 (+ 12254us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:43:58.023980 (+ 5651us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:43:58.023985 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:43:58.023989 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:43:58.024498 (+ 509us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:43:58.025319 (+ 821us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:43:58.025324 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:43:58.025328 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:43:58.025873 (+ 545us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:43:58.026641 (+ 768us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:43:58.026646 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:43:58.026802 (+ 156us) client_negotiation.cc:769] Sending connection context
0619 16:43:58.026948 (+ 146us) client_negotiation.cc:240] Negotiation successful
0619 16:43:58.030982 (+ 4034us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14132,"mutex_wait_us":3550,"spinlock_wait_cycles":22912,"thread_start_us":14217,"threads_started":1}
I20260619 16:43:58.033344 7786 heartbeater.cc:347] Connected to a master server at 127.0.97.62:46817
I20260619 16:43:58.033339 7787 heartbeater.cc:347] Connected to a master server at 127.0.97.61:42337
I20260619 16:43:58.033661 7786 heartbeater.cc:464] Registering TS with master...
I20260619 16:43:58.033675 7787 heartbeater.cc:464] Registering TS with master...
I20260619 16:43:58.033308 7785 heartbeater.cc:347] Connected to a master server at 127.0.97.60:35535
I20260619 16:43:58.033968 7785 heartbeater.cc:464] Registering TS with master...
I20260619 16:43:58.034581 7785 heartbeater.cc:511] Master 127.0.97.60:35535 requested a full tablet report, sending...
I20260619 16:43:58.035513 7786 heartbeater.cc:511] Master 127.0.97.62:46817 requested a full tablet report, sending...
I20260619 16:43:58.036156 7787 heartbeater.cc:511] Master 127.0.97.61:42337 requested a full tablet report, sending...
I20260619 16:43:58.036319 7314 ts_manager.cc:194] Registered new tserver with Master: 399a36acb0b447e8b9c0d6992bc86ea2 (127.0.97.3:40501)
I20260619 16:43:58.037341 7124 ts_manager.cc:194] Registered new tserver with Master: 399a36acb0b447e8b9c0d6992bc86ea2 (127.0.97.3:40501)
I20260619 16:43:58.038427 6527 ts_manager.cc:194] Re-registered known tserver with Master: 399a36acb0b447e8b9c0d6992bc86ea2 (127.0.97.3:40501)
I20260619 16:43:58.039453 6527 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:35909
I20260619 16:43:58.050170 7795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:58.008332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46104 (local address 127.0.97.62:46817)
0619 16:43:58.030523 (+ 22191us) server_negotiation.cc:206] Beginning negotiation
0619 16:43:58.030528 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:43:58.033378 (+ 2850us) server_negotiation.cc:407] Connection header received
0619 16:43:58.033430 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:43:58.033434 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:43:58.033499 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:43:58.033582 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:43:58.036662 (+ 3080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:58.037651 (+ 989us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:58.039817 (+ 2166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:58.040189 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:58.040293 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:43:58.043738 (+ 3445us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:43:58.043744 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:43:58.043750 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:43:58.043786 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:43:58.046827 (+ 3041us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:43:58.047897 (+ 1070us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:43:58.047903 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:43:58.047906 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:43:58.048034 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:43:58.048918 (+ 884us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:43:58.048921 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:43:58.048923 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:43:58.049158 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:43:58.049363 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0619 16:43:58.049666 (+ 303us) server_negotiation.cc:299] Negotiation successful
0619 16:43:58.049891 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22104,"thread_start_us":20712,"threads_started":1}
I20260619 16:43:58.052848 7790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:58.009292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42146 (local address 127.0.97.61:42337)
0619 16:43:58.028210 (+ 18918us) server_negotiation.cc:206] Beginning negotiation
0619 16:43:58.028216 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:43:58.037374 (+ 9158us) server_negotiation.cc:407] Connection header received
0619 16:43:58.037590 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:43:58.037595 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:43:58.037681 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:43:58.037772 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:43:58.041126 (+ 3354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:58.042195 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:58.043951 (+ 1756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:58.044344 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:58.044456 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:43:58.047388 (+ 2932us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:43:58.047394 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:43:58.047398 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:43:58.047437 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:43:58.050244 (+ 2807us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:43:58.050936 (+ 692us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:43:58.050940 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:43:58.050944 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:43:58.051076 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:43:58.051424 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:43:58.051429 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:43:58.051433 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:43:58.051743 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:43:58.051956 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0619 16:43:58.052336 (+ 380us) server_negotiation.cc:299] Negotiation successful
0619 16:43:58.052541 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18811,"thread_start_us":19538,"threads_started":1}
I20260619 16:43:58.054162 7789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:58.037294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55128 (local address 127.0.97.60:35535)
0619 16:43:58.038462 (+ 1168us) server_negotiation.cc:206] Beginning negotiation
0619 16:43:58.038468 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:43:58.039116 (+ 648us) server_negotiation.cc:407] Connection header received
0619 16:43:58.039342 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:43:58.039347 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:43:58.039441 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:43:58.039504 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:43:58.042019 (+ 2515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:58.043010 (+ 991us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:58.044676 (+ 1666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:58.045776 (+ 1100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:58.045892 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:43:58.048403 (+ 2511us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:43:58.048409 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:43:58.048414 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:43:58.048446 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:43:58.051377 (+ 2931us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:43:58.052067 (+ 690us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:43:58.052072 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:43:58.052076 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:43:58.052193 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:43:58.052497 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:43:58.052502 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:43:58.052506 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:43:58.052747 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:43:58.053461 (+ 714us) server_negotiation.cc:1035] Waiting for connection context
0619 16:43:58.053807 (+ 346us) server_negotiation.cc:299] Negotiation successful
0619 16:43:58.053956 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1046}
I20260619 16:43:58.054020 6527 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46104:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 16:43:58.060925 6527 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 16:43:58.109277 7813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:58.076992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46546 (local address 127.0.97.2:34331)
0619 16:43:58.101950 (+ 24958us) server_negotiation.cc:206] Beginning negotiation
0619 16:43:58.101958 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 16:43:58.101987 (+ 29us) server_negotiation.cc:407] Connection header received
0619 16:43:58.102080 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:43:58.102089 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:43:58.102386 (+ 297us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:43:58.102981 (+ 595us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:43:58.103428 (+ 447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:58.105217 (+ 1789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:58.107000 (+ 1783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:58.108325 (+ 1325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:58.108445 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:43:58.108638 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0619 16:43:58.108852 (+ 214us) server_negotiation.cc:299] Negotiation successful
0619 16:43:58.109008 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24805,"thread_start_us":24666,"threads_started":1}
I20260619 16:43:58.109684 7815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:58.076740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:34331 (local address 127.0.0.1:46546)
0619 16:43:58.099092 (+ 22352us) negotiation.cc:107] Waiting for socket to connect
0619 16:43:58.099113 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 16:43:58.099238 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:43:58.102606 (+ 3368us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:43:58.102614 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:43:58.102664 (+ 50us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:43:58.103095 (+ 431us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:43:58.103105 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:43:58.105331 (+ 2226us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:43:58.105335 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:43:58.106881 (+ 1546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:43:58.106895 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:43:58.108511 (+ 1616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:43:58.108517 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:43:58.108675 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:43:58.108710 (+ 35us) client_negotiation.cc:769] Sending connection context
0619 16:43:58.108806 (+ 96us) client_negotiation.cc:240] Negotiation successful
0619 16:43:58.108932 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":22220,"spinlock_wait_cycles":768,"thread_start_us":24376,"threads_started":1}
I20260619 16:43:58.111431 7814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:58.077562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:40501 (local address 127.0.0.1:51222)
0619 16:43:58.103461 (+ 25899us) negotiation.cc:107] Waiting for socket to connect
0619 16:43:58.103476 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 16:43:58.103586 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:43:58.104104 (+ 518us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:43:58.104108 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:43:58.104131 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:43:58.104402 (+ 271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:43:58.104410 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:43:58.107047 (+ 2637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:43:58.107051 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:43:58.109284 (+ 2233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:43:58.109300 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:43:58.110925 (+ 1625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:43:58.110929 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:43:58.111038 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:43:58.111057 (+ 19us) client_negotiation.cc:769] Sending connection context
0619 16:43:58.111120 (+ 63us) client_negotiation.cc:240] Negotiation successful
0619 16:43:58.111201 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25804,"mutex_wait_us":2,"thread_start_us":22274,"threads_started":1}
I20260619 16:43:58.111616 7817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:58.078171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51222 (local address 127.0.97.3:40501)
0619 16:43:58.099806 (+ 21635us) server_negotiation.cc:206] Beginning negotiation
0619 16:43:58.099817 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 16:43:58.103580 (+ 3763us) server_negotiation.cc:407] Connection header received
0619 16:43:58.103719 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:43:58.103726 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:43:58.103917 (+ 191us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:43:58.104095 (+ 178us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:43:58.104572 (+ 477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:58.106569 (+ 1997us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:58.109462 (+ 2893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:58.110587 (+ 1125us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:58.110657 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:43:58.110796 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0619 16:43:58.111185 (+ 389us) server_negotiation.cc:299] Negotiation successful
0619 16:43:58.111334 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21447,"thread_start_us":22607,"threads_started":1}
I20260619 16:43:58.113296 7571 tablet_service.cc:1467] Processing CreateTablet for tablet aec7d89f1a054af29ed83f30837f7c41 (DEFAULT_TABLE table=test-table [id=6e015cbc439c499bb035b38a4d48fc44]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:43:58.114403 7571 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aec7d89f1a054af29ed83f30837f7c41. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:43:58.115605 7720 tablet_service.cc:1467] Processing CreateTablet for tablet aec7d89f1a054af29ed83f30837f7c41 (DEFAULT_TABLE table=test-table [id=6e015cbc439c499bb035b38a4d48fc44]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:43:58.117131 7812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:58.077154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:40321 (local address 127.0.0.1:33746)
0619 16:43:58.101378 (+ 24224us) negotiation.cc:107] Waiting for socket to connect
0619 16:43:58.101400 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0619 16:43:58.101508 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:43:58.102550 (+ 1042us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:43:58.102555 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:43:58.102578 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:43:58.103070 (+ 492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:43:58.103087 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:43:58.106938 (+ 3851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:43:58.106943 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:43:58.109313 (+ 2370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:43:58.109330 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:43:58.116372 (+ 7042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:43:58.116379 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:43:58.116534 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:43:58.116565 (+ 31us) client_negotiation.cc:769] Sending connection context
0619 16:43:58.116657 (+ 92us) client_negotiation.cc:240] Negotiation successful
0619 16:43:58.116790 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24094,"mutex_wait_us":68,"thread_start_us":26095,"threads_started":1}
I20260619 16:43:58.117180 7720 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aec7d89f1a054af29ed83f30837f7c41. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:43:58.117245 7816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:58.077599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33746 (local address 127.0.97.1:40321)
0619 16:43:58.102107 (+ 24508us) server_negotiation.cc:206] Beginning negotiation
0619 16:43:58.102113 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:43:58.102135 (+ 22us) server_negotiation.cc:407] Connection header received
0619 16:43:58.102211 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:43:58.102216 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:43:58.102372 (+ 156us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:43:58.102550 (+ 178us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:43:58.105409 (+ 2859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:58.106823 (+ 1414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:58.109478 (+ 2655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:58.110742 (+ 1264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:58.116272 (+ 5530us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:43:58.116464 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0619 16:43:58.116728 (+ 264us) server_negotiation.cc:299] Negotiation successful
0619 16:43:58.116908 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24377,"thread_start_us":24293,"threads_started":1}
I20260619 16:43:58.121098 7428 tablet_service.cc:1467] Processing CreateTablet for tablet aec7d89f1a054af29ed83f30837f7c41 (DEFAULT_TABLE table=test-table [id=6e015cbc439c499bb035b38a4d48fc44]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:43:58.122335 7428 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aec7d89f1a054af29ed83f30837f7c41. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:43:58.125919 7818 tablet_bootstrap.cc:492] T aec7d89f1a054af29ed83f30837f7c41 P 449f7e56e3754b22a114d8f19ed82f76: Bootstrap starting.
I20260619 16:43:58.128571 7818 tablet_bootstrap.cc:654] T aec7d89f1a054af29ed83f30837f7c41 P 449f7e56e3754b22a114d8f19ed82f76: Neither blocks nor log segments found. Creating new log.
I20260619 16:43:58.129448 7818 log.cc:824] T aec7d89f1a054af29ed83f30837f7c41 P 449f7e56e3754b22a114d8f19ed82f76: Log is configured to *not* fsync() on all Append() calls
I20260619 16:43:58.133365 7820 tablet_bootstrap.cc:492] T aec7d89f1a054af29ed83f30837f7c41 P 399a36acb0b447e8b9c0d6992bc86ea2: Bootstrap starting.
I20260619 16:43:58.135738 7818 tablet_bootstrap.cc:492] T aec7d89f1a054af29ed83f30837f7c41 P 449f7e56e3754b22a114d8f19ed82f76: No bootstrap required, opened a new log
I20260619 16:43:58.135987 7818 ts_tablet_manager.cc:1397] T aec7d89f1a054af29ed83f30837f7c41 P 449f7e56e3754b22a114d8f19ed82f76: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260619 16:43:58.137094 7821 tablet_bootstrap.cc:492] T aec7d89f1a054af29ed83f30837f7c41 P 4e87af70e6654dff8d6f88b6b739ee42: Bootstrap starting.
I20260619 16:43:58.137140 7820 tablet_bootstrap.cc:654] T aec7d89f1a054af29ed83f30837f7c41 P 399a36acb0b447e8b9c0d6992bc86ea2: Neither blocks nor log segments found. Creating new log.
I20260619 16:43:58.138550 7820 log.cc:824] T aec7d89f1a054af29ed83f30837f7c41 P 399a36acb0b447e8b9c0d6992bc86ea2: Log is configured to *not* fsync() on all Append() calls
I20260619 16:43:58.140218 7821 tablet_bootstrap.cc:654] T aec7d89f1a054af29ed83f30837f7c41 P 4e87af70e6654dff8d6f88b6b739ee42: Neither blocks nor log segments found. Creating new log.
I20260619 16:43:58.141124 7821 log.cc:824] T aec7d89f1a054af29ed83f30837f7c41 P 4e87af70e6654dff8d6f88b6b739ee42: Log is configured to *not* fsync() on all Append() calls
I20260619 16:43:58.141340 7820 tablet_bootstrap.cc:492] T aec7d89f1a054af29ed83f30837f7c41 P 399a36acb0b447e8b9c0d6992bc86ea2: No bootstrap required, opened a new log
I20260619 16:43:58.141569 7820 ts_tablet_manager.cc:1397] T aec7d89f1a054af29ed83f30837f7c41 P 399a36acb0b447e8b9c0d6992bc86ea2: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.003s
I20260619 16:43:58.143391 7821 tablet_bootstrap.cc:492] T aec7d89f1a054af29ed83f30837f7c41 P 4e87af70e6654dff8d6f88b6b739ee42: No bootstrap required, opened a new log
I20260619 16:43:58.143610 7821 ts_tablet_manager.cc:1397] T aec7d89f1a054af29ed83f30837f7c41 P 4e87af70e6654dff8d6f88b6b739ee42: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260619 16:43:58.145896 7818 raft_consensus.cc:348] T aec7d89f1a054af29ed83f30837f7c41 P 449f7e56e3754b22a114d8f19ed82f76 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "449f7e56e3754b22a114d8f19ed82f76" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34331 } } peers { permanent_uuid: "4e87af70e6654dff8d6f88b6b739ee42" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40321 } } peers { permanent_uuid: "399a36acb0b447e8b9c0d6992bc86ea2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40501 } }
I20260619 16:43:58.146417 7818 raft_consensus.cc:374] T aec7d89f1a054af29ed83f30837f7c41 P 449f7e56e3754b22a114d8f19ed82f76 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:43:58.146514 7818 raft_consensus.cc:729] T aec7d89f1a054af29ed83f30837f7c41 P 449f7e56e3754b22a114d8f19ed82f76 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 449f7e56e3754b22a114d8f19ed82f76, State: Initialized, Role: FOLLOWER
I20260619 16:43:58.147075 7818 consensus_queue.cc:260] T aec7d89f1a054af29ed83f30837f7c41 P 449f7e56e3754b22a114d8f19ed82f76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "449f7e56e3754b22a114d8f19ed82f76" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34331 } } peers { permanent_uuid: "4e87af70e6654dff8d6f88b6b739ee42" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40321 } } peers { permanent_uuid: "399a36acb0b447e8b9c0d6992bc86ea2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40501 } }
I20260619 16:43:58.148541 7818 ts_tablet_manager.cc:1428] T aec7d89f1a054af29ed83f30837f7c41 P 449f7e56e3754b22a114d8f19ed82f76: Time spent starting tablet: real 0.012s user 0.013s sys 0.000s
I20260619 16:43:58.148824 7637 heartbeater.cc:503] Master 127.0.97.62:46817 was elected leader, sending a full tablet report...
I20260619 16:43:58.151933 7820 raft_consensus.cc:348] T aec7d89f1a054af29ed83f30837f7c41 P 399a36acb0b447e8b9c0d6992bc86ea2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "449f7e56e3754b22a114d8f19ed82f76" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34331 } } peers { permanent_uuid: "4e87af70e6654dff8d6f88b6b739ee42" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40321 } } peers { permanent_uuid: "399a36acb0b447e8b9c0d6992bc86ea2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40501 } }
I20260619 16:43:58.152321 7820 raft_consensus.cc:374] T aec7d89f1a054af29ed83f30837f7c41 P 399a36acb0b447e8b9c0d6992bc86ea2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:43:58.152441 7820 raft_consensus.cc:729] T aec7d89f1a054af29ed83f30837f7c41 P 399a36acb0b447e8b9c0d6992bc86ea2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 399a36acb0b447e8b9c0d6992bc86ea2, State: Initialized, Role: FOLLOWER
I20260619 16:43:58.152977 7820 consensus_queue.cc:260] T aec7d89f1a054af29ed83f30837f7c41 P 399a36acb0b447e8b9c0d6992bc86ea2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "449f7e56e3754b22a114d8f19ed82f76" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34331 } } peers { permanent_uuid: "4e87af70e6654dff8d6f88b6b739ee42" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40321 } } peers { permanent_uuid: "399a36acb0b447e8b9c0d6992bc86ea2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40501 } }
I20260619 16:43:58.153955 7786 heartbeater.cc:503] Master 127.0.97.62:46817 was elected leader, sending a full tablet report...
I20260619 16:43:58.154605 7820 ts_tablet_manager.cc:1428] T aec7d89f1a054af29ed83f30837f7c41 P 399a36acb0b447e8b9c0d6992bc86ea2: Time spent starting tablet: real 0.013s user 0.011s sys 0.001s
I20260619 16:43:58.155635 7821 raft_consensus.cc:348] T aec7d89f1a054af29ed83f30837f7c41 P 4e87af70e6654dff8d6f88b6b739ee42 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "449f7e56e3754b22a114d8f19ed82f76" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34331 } } peers { permanent_uuid: "4e87af70e6654dff8d6f88b6b739ee42" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40321 } } peers { permanent_uuid: "399a36acb0b447e8b9c0d6992bc86ea2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40501 } }
I20260619 16:43:58.156020 7821 raft_consensus.cc:374] T aec7d89f1a054af29ed83f30837f7c41 P 4e87af70e6654dff8d6f88b6b739ee42 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:43:58.156116 7821 raft_consensus.cc:729] T aec7d89f1a054af29ed83f30837f7c41 P 4e87af70e6654dff8d6f88b6b739ee42 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e87af70e6654dff8d6f88b6b739ee42, State: Initialized, Role: FOLLOWER
I20260619 16:43:58.156631 7821 consensus_queue.cc:260] T aec7d89f1a054af29ed83f30837f7c41 P 4e87af70e6654dff8d6f88b6b739ee42 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "449f7e56e3754b22a114d8f19ed82f76" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34331 } } peers { permanent_uuid: "4e87af70e6654dff8d6f88b6b739ee42" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40321 } } peers { permanent_uuid: "399a36acb0b447e8b9c0d6992bc86ea2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40501 } }
I20260619 16:43:58.159690 7821 ts_tablet_manager.cc:1428] T aec7d89f1a054af29ed83f30837f7c41 P 4e87af70e6654dff8d6f88b6b739ee42: Time spent starting tablet: real 0.016s user 0.008s sys 0.006s
W20260619 16:43:58.201074 7645 tablet.cc:2369] T aec7d89f1a054af29ed83f30837f7c41 P 449f7e56e3754b22a114d8f19ed82f76: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 16:43:58.233995 7788 tablet.cc:2369] T aec7d89f1a054af29ed83f30837f7c41 P 399a36acb0b447e8b9c0d6992bc86ea2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 16:43:58.303558 7826 raft_consensus.cc:482] T aec7d89f1a054af29ed83f30837f7c41 P 4e87af70e6654dff8d6f88b6b739ee42 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 16:43:58.303860 7826 raft_consensus.cc:504] T aec7d89f1a054af29ed83f30837f7c41 P 4e87af70e6654dff8d6f88b6b739ee42 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "449f7e56e3754b22a114d8f19ed82f76" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34331 } } peers { permanent_uuid: "4e87af70e6654dff8d6f88b6b739ee42" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40321 } } peers { permanent_uuid: "399a36acb0b447e8b9c0d6992bc86ea2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40501 } }
I20260619 16:43:58.305456 7826 leader_election.cc:288] T aec7d89f1a054af29ed83f30837f7c41 P 4e87af70e6654dff8d6f88b6b739ee42 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 449f7e56e3754b22a114d8f19ed82f76 (127.0.97.2:34331), 399a36acb0b447e8b9c0d6992bc86ea2 (127.0.97.3:40501)
I20260619 16:43:58.312584 7813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:58.305788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:38215 (local address 127.0.97.2:34331)
0619 16:43:58.305984 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0619 16:43:58.305991 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:43:58.306526 (+ 535us) server_negotiation.cc:407] Connection header received
0619 16:43:58.306678 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:43:58.306684 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:43:58.306767 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:43:58.306870 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:43:58.307547 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:58.308296 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:58.310307 (+ 2011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:58.311825 (+ 1518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:58.311920 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:43:58.311998 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0619 16:43:58.312323 (+ 325us) server_negotiation.cc:299] Negotiation successful
0619 16:43:58.312411 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260619 16:43:58.312788 7827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:58.305654 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:34331 (local address 127.0.97.1:38215)
0619 16:43:58.306420 (+ 766us) negotiation.cc:107] Waiting for socket to connect
0619 16:43:58.306437 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 16:43:58.306560 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:43:58.306881 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:43:58.306884 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:43:58.306902 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:43:58.307232 (+ 330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:43:58.307250 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:43:58.308397 (+ 1147us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:43:58.308402 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:43:58.310146 (+ 1744us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:43:58.310164 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:43:58.312007 (+ 1843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:43:58.312014 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:43:58.312180 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:43:58.312215 (+ 35us) client_negotiation.cc:769] Sending connection context
0619 16:43:58.312302 (+ 87us) client_negotiation.cc:240] Negotiation successful
0619 16:43:58.312424 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":656,"mutex_wait_us":1,"thread_start_us":352,"threads_started":1}
I20260619 16:43:58.313630 7817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:58.306126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:60603 (local address 127.0.97.3:40501)
0619 16:43:58.306379 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0619 16:43:58.306385 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:43:58.307541 (+ 1156us) server_negotiation.cc:407] Connection header received
0619 16:43:58.307686 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:43:58.307690 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:43:58.307765 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:43:58.307842 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:43:58.308258 (+ 416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:58.309257 (+ 999us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:58.311820 (+ 2563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:58.312903 (+ 1083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:58.313016 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:43:58.313113 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0619 16:43:58.313379 (+ 266us) server_negotiation.cc:299] Negotiation successful
0619 16:43:58.313469 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260619 16:43:58.313787 7828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:58.305941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:40501 (local address 127.0.97.1:60603)
0619 16:43:58.307443 (+ 1502us) negotiation.cc:107] Waiting for socket to connect
0619 16:43:58.307457 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 16:43:58.307566 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:43:58.307857 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:43:58.307861 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:43:58.307882 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:43:58.308121 (+ 239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:43:58.308129 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:43:58.309378 (+ 1249us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:43:58.309382 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:43:58.311278 (+ 1896us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:43:58.311293 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:43:58.313080 (+ 1787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:43:58.313085 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:43:58.313231 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:43:58.313252 (+ 21us) client_negotiation.cc:769] Sending connection context
0619 16:43:58.313334 (+ 82us) client_negotiation.cc:240] Negotiation successful
0619 16:43:58.313447 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1386,"mutex_wait_us":21,"thread_start_us":1234,"threads_started":1}
I20260619 16:43:58.313797 7591 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aec7d89f1a054af29ed83f30837f7c41" candidate_uuid: "4e87af70e6654dff8d6f88b6b739ee42" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "449f7e56e3754b22a114d8f19ed82f76" is_pre_election: true
I20260619 16:43:58.314321 7591 raft_consensus.cc:2454] T aec7d89f1a054af29ed83f30837f7c41 P 449f7e56e3754b22a114d8f19ed82f76 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4e87af70e6654dff8d6f88b6b739ee42 in term 0.
I20260619 16:43:58.314734 7740 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aec7d89f1a054af29ed83f30837f7c41" candidate_uuid: "4e87af70e6654dff8d6f88b6b739ee42" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "399a36acb0b447e8b9c0d6992bc86ea2" is_pre_election: true
I20260619 16:43:58.314931 7382 leader_election.cc:302] T aec7d89f1a054af29ed83f30837f7c41 P 4e87af70e6654dff8d6f88b6b739ee42 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 449f7e56e3754b22a114d8f19ed82f76, 4e87af70e6654dff8d6f88b6b739ee42; no voters:
I20260619 16:43:58.315291 7826 raft_consensus.cc:2781] T aec7d89f1a054af29ed83f30837f7c41 P 4e87af70e6654dff8d6f88b6b739ee42 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 16:43:58.315255 7740 raft_consensus.cc:2454] T aec7d89f1a054af29ed83f30837f7c41 P 399a36acb0b447e8b9c0d6992bc86ea2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4e87af70e6654dff8d6f88b6b739ee42 in term 0.
I20260619 16:43:58.315400 7826 raft_consensus.cc:482] T aec7d89f1a054af29ed83f30837f7c41 P 4e87af70e6654dff8d6f88b6b739ee42 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 16:43:58.315481 7826 raft_consensus.cc:3037] T aec7d89f1a054af29ed83f30837f7c41 P 4e87af70e6654dff8d6f88b6b739ee42 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:43:58.317595 7826 raft_consensus.cc:504] T aec7d89f1a054af29ed83f30837f7c41 P 4e87af70e6654dff8d6f88b6b739ee42 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "449f7e56e3754b22a114d8f19ed82f76" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34331 } } peers { permanent_uuid: "4e87af70e6654dff8d6f88b6b739ee42" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40321 } } peers { permanent_uuid: "399a36acb0b447e8b9c0d6992bc86ea2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40501 } }
I20260619 16:43:58.318166 7826 leader_election.cc:288] T aec7d89f1a054af29ed83f30837f7c41 P 4e87af70e6654dff8d6f88b6b739ee42 [CANDIDATE]: Term 1 election: Requested vote from peers 449f7e56e3754b22a114d8f19ed82f76 (127.0.97.2:34331), 399a36acb0b447e8b9c0d6992bc86ea2 (127.0.97.3:40501)
I20260619 16:43:58.318509 7591 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aec7d89f1a054af29ed83f30837f7c41" candidate_uuid: "4e87af70e6654dff8d6f88b6b739ee42" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "449f7e56e3754b22a114d8f19ed82f76"
I20260619 16:43:58.318573 7740 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aec7d89f1a054af29ed83f30837f7c41" candidate_uuid: "4e87af70e6654dff8d6f88b6b739ee42" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "399a36acb0b447e8b9c0d6992bc86ea2"
I20260619 16:43:58.318663 7591 raft_consensus.cc:3037] T aec7d89f1a054af29ed83f30837f7c41 P 449f7e56e3754b22a114d8f19ed82f76 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:43:58.318789 7740 raft_consensus.cc:3037] T aec7d89f1a054af29ed83f30837f7c41 P 399a36acb0b447e8b9c0d6992bc86ea2 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:43:58.320694 7591 raft_consensus.cc:2454] T aec7d89f1a054af29ed83f30837f7c41 P 449f7e56e3754b22a114d8f19ed82f76 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e87af70e6654dff8d6f88b6b739ee42 in term 1.
I20260619 16:43:58.320994 7382 leader_election.cc:302] T aec7d89f1a054af29ed83f30837f7c41 P 4e87af70e6654dff8d6f88b6b739ee42 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 449f7e56e3754b22a114d8f19ed82f76, 4e87af70e6654dff8d6f88b6b739ee42; no voters:
I20260619 16:43:58.321233 7826 raft_consensus.cc:2781] T aec7d89f1a054af29ed83f30837f7c41 P 4e87af70e6654dff8d6f88b6b739ee42 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:43:58.321393 7740 raft_consensus.cc:2454] T aec7d89f1a054af29ed83f30837f7c41 P 399a36acb0b447e8b9c0d6992bc86ea2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e87af70e6654dff8d6f88b6b739ee42 in term 1.
I20260619 16:43:58.321662 7826 raft_consensus.cc:686] T aec7d89f1a054af29ed83f30837f7c41 P 4e87af70e6654dff8d6f88b6b739ee42 [term 1 LEADER]: Becoming Leader. State: Replica: 4e87af70e6654dff8d6f88b6b739ee42, State: Running, Role: LEADER
I20260619 16:43:58.322211 7826 consensus_queue.cc:237] T aec7d89f1a054af29ed83f30837f7c41 P 4e87af70e6654dff8d6f88b6b739ee42 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "449f7e56e3754b22a114d8f19ed82f76" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34331 } } peers { permanent_uuid: "4e87af70e6654dff8d6f88b6b739ee42" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40321 } } peers { permanent_uuid: "399a36acb0b447e8b9c0d6992bc86ea2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40501 } }
I20260619 16:43:58.326704 6527 catalog_manager.cc:5495] T aec7d89f1a054af29ed83f30837f7c41 P 4e87af70e6654dff8d6f88b6b739ee42 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4e87af70e6654dff8d6f88b6b739ee42 (127.0.97.1). New cstate: current_term: 1 leader_uuid: "4e87af70e6654dff8d6f88b6b739ee42" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "449f7e56e3754b22a114d8f19ed82f76" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34331 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4e87af70e6654dff8d6f88b6b739ee42" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40321 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "399a36acb0b447e8b9c0d6992bc86ea2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40501 } health_report { overall_health: UNKNOWN } } }
W20260619 16:43:58.366425 7496 tablet.cc:2369] T aec7d89f1a054af29ed83f30837f7c41 P 4e87af70e6654dff8d6f88b6b739ee42: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 16:43:58.398427 7816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:58.393403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33754 (local address 127.0.97.1:40321)
0619 16:43:58.393588 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0619 16:43:58.393593 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:43:58.393615 (+ 22us) server_negotiation.cc:407] Connection header received
0619 16:43:58.393711 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:43:58.393716 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:43:58.393833 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:43:58.393978 (+ 145us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 16:43:58.394310 (+ 332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:58.394985 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:58.396271 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:58.396794 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:58.396908 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:43:58.397355 (+ 447us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 16:43:58.397872 (+ 517us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 16:43:58.398004 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0619 16:43:58.398091 (+ 87us) server_negotiation.cc:299] Negotiation successful
0619 16:43:58.398167 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260619 16:43:58.406106 7740 raft_consensus.cc:1261] T aec7d89f1a054af29ed83f30837f7c41 P 399a36acb0b447e8b9c0d6992bc86ea2 [term 1 FOLLOWER]: Refusing update from remote peer 4e87af70e6654dff8d6f88b6b739ee42: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:43:58.406131 7591 raft_consensus.cc:1261] T aec7d89f1a054af29ed83f30837f7c41 P 449f7e56e3754b22a114d8f19ed82f76 [term 1 FOLLOWER]: Refusing update from remote peer 4e87af70e6654dff8d6f88b6b739ee42: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:43:58.406950 7826 consensus_queue.cc:1035] T aec7d89f1a054af29ed83f30837f7c41 P 4e87af70e6654dff8d6f88b6b739ee42 [LEADER]: Connected to new peer: Peer: permanent_uuid: "399a36acb0b447e8b9c0d6992bc86ea2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:43:58.407202 7829 consensus_queue.cc:1035] T aec7d89f1a054af29ed83f30837f7c41 P 4e87af70e6654dff8d6f88b6b739ee42 [LEADER]: Connected to new peer: Peer: permanent_uuid: "449f7e56e3754b22a114d8f19ed82f76" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34331 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:43:58.419821 7832 mvcc.cc:205] Tried to move back new op lower bound from 7298610947702960128 to 7298610947371024384. Current Snapshot: MvccSnapshot[applied={T|T < 7298610947702960128}]
I20260619 16:43:58.424669 7834 mvcc.cc:205] Tried to move back new op lower bound from 7298610947702960128 to 7298610947371024384. Current Snapshot: MvccSnapshot[applied={T|T < 7298610947702960128}]
I20260619 16:43:58.447291 6527 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46104:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:43:58.447573 6527 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46104:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:43:58.457567 6527 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5ba5d12f1093420e96661d1c478add9b: Sending DeleteTablet for 3 replicas of tablet aec7d89f1a054af29ed83f30837f7c41
I20260619 16:43:58.458776 7720 tablet_service.cc:1514] Processing DeleteTablet for tablet aec7d89f1a054af29ed83f30837f7c41 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:43:58 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:51222
I20260619 16:43:58.458961 7571 tablet_service.cc:1514] Processing DeleteTablet for tablet aec7d89f1a054af29ed83f30837f7c41 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:43:58 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:46546
I20260619 16:43:58.459825 7428 tablet_service.cc:1514] Processing DeleteTablet for tablet aec7d89f1a054af29ed83f30837f7c41 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:43:58 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:33746
I20260619 16:43:58.460072 7841 tablet_replica.cc:330] T aec7d89f1a054af29ed83f30837f7c41 P 399a36acb0b447e8b9c0d6992bc86ea2: stopping tablet replica
I20260619 16:43:58.460462 7841 raft_consensus.cc:2229] T aec7d89f1a054af29ed83f30837f7c41 P 399a36acb0b447e8b9c0d6992bc86ea2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 16:43:58.460943 7841 raft_consensus.cc:2258] T aec7d89f1a054af29ed83f30837f7c41 P 399a36acb0b447e8b9c0d6992bc86ea2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:43:58.469246 7841 ts_tablet_manager.cc:1905] T aec7d89f1a054af29ed83f30837f7c41 P 399a36acb0b447e8b9c0d6992bc86ea2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:43:58.470043 7843 tablet_replica.cc:330] T aec7d89f1a054af29ed83f30837f7c41 P 4e87af70e6654dff8d6f88b6b739ee42: stopping tablet replica
I20260619 16:43:58.471216 7843 raft_consensus.cc:2229] T aec7d89f1a054af29ed83f30837f7c41 P 4e87af70e6654dff8d6f88b6b739ee42 [term 1 LEADER]: Raft consensus shutting down.
I20260619 16:43:58.473356 7842 tablet_replica.cc:330] T aec7d89f1a054af29ed83f30837f7c41 P 449f7e56e3754b22a114d8f19ed82f76: stopping tablet replica
I20260619 16:43:58.473693 7842 raft_consensus.cc:2229] T aec7d89f1a054af29ed83f30837f7c41 P 449f7e56e3754b22a114d8f19ed82f76 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 16:43:58.474030 7843 raft_consensus.cc:2258] T aec7d89f1a054af29ed83f30837f7c41 P 4e87af70e6654dff8d6f88b6b739ee42 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:43:58.474792 7842 raft_consensus.cc:2258] T aec7d89f1a054af29ed83f30837f7c41 P 449f7e56e3754b22a114d8f19ed82f76 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:43:58.476333 7843 ts_tablet_manager.cc:1905] T aec7d89f1a054af29ed83f30837f7c41 P 4e87af70e6654dff8d6f88b6b739ee42: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:43:58.477029 7842 ts_tablet_manager.cc:1905] T aec7d89f1a054af29ed83f30837f7c41 P 449f7e56e3754b22a114d8f19ed82f76: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:43:58.477675 7841 ts_tablet_manager.cc:1918] T aec7d89f1a054af29ed83f30837f7c41 P 399a36acb0b447e8b9c0d6992bc86ea2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 16:43:58.477888 7841 log.cc:1196] T aec7d89f1a054af29ed83f30837f7c41 P 399a36acb0b447e8b9c0d6992bc86ea2: Deleting WAL directory at /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887385903558-388-0/minicluster-data/ts-2/wal/wals/aec7d89f1a054af29ed83f30837f7c41
I20260619 16:43:58.478717 7841 ts_tablet_manager.cc:1939] T aec7d89f1a054af29ed83f30837f7c41 P 399a36acb0b447e8b9c0d6992bc86ea2: Deleting consensus metadata
I20260619 16:43:58.480417 6511 catalog_manager.cc:4841] TS 399a36acb0b447e8b9c0d6992bc86ea2 (127.0.97.3:40501): tablet aec7d89f1a054af29ed83f30837f7c41 (table test-table [id=6e015cbc439c499bb035b38a4d48fc44]) successfully deleted
I20260619 16:43:58.483867 7842 ts_tablet_manager.cc:1918] T aec7d89f1a054af29ed83f30837f7c41 P 449f7e56e3754b22a114d8f19ed82f76: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 16:43:58.484366 7843 ts_tablet_manager.cc:1918] T aec7d89f1a054af29ed83f30837f7c41 P 4e87af70e6654dff8d6f88b6b739ee42: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 16:43:58.484649 7843 log.cc:1196] T aec7d89f1a054af29ed83f30837f7c41 P 4e87af70e6654dff8d6f88b6b739ee42: Deleting WAL directory at /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887385903558-388-0/minicluster-data/ts-0/wal/wals/aec7d89f1a054af29ed83f30837f7c41
I20260619 16:43:58.485463 7842 log.cc:1196] T aec7d89f1a054af29ed83f30837f7c41 P 449f7e56e3754b22a114d8f19ed82f76: Deleting WAL directory at /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887385903558-388-0/minicluster-data/ts-1/wal/wals/aec7d89f1a054af29ed83f30837f7c41
I20260619 16:43:58.486344 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu pid 6495
I20260619 16:43:58.486518 7843 ts_tablet_manager.cc:1939] T aec7d89f1a054af29ed83f30837f7c41 P 4e87af70e6654dff8d6f88b6b739ee42: Deleting consensus metadata
I20260619 16:43:58.487991 6514 catalog_manager.cc:4841] TS 4e87af70e6654dff8d6f88b6b739ee42 (127.0.97.1:40321): tablet aec7d89f1a054af29ed83f30837f7c41 (table test-table [id=6e015cbc439c499bb035b38a4d48fc44]) successfully deleted
I20260619 16:43:58.488301 7842 ts_tablet_manager.cc:1939] T aec7d89f1a054af29ed83f30837f7c41 P 449f7e56e3754b22a114d8f19ed82f76: Deleting consensus metadata
I20260619 16:43:58.488435 6557 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:43:58.502586 7790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:58.463267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42152 (local address 127.0.97.61:42337)
0619 16:43:58.463443 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0619 16:43:58.463449 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:43:58.485142 (+ 21693us) server_negotiation.cc:407] Connection header received
0619 16:43:58.485373 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:43:58.485379 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:43:58.485461 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:43:58.485565 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:43:58.488497 (+ 2932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:58.489690 (+ 1193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:58.491915 (+ 2225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:58.492340 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:58.492432 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:43:58.496298 (+ 3866us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:43:58.496305 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:43:58.496310 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:43:58.496348 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:43:58.499769 (+ 3421us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:43:58.500537 (+ 768us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:43:58.500542 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:43:58.500546 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:43:58.500684 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:43:58.501067 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:43:58.501072 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:43:58.501075 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:43:58.501439 (+ 364us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:43:58.501715 (+ 276us) server_negotiation.cc:1035] Waiting for connection context
0619 16:43:58.502059 (+ 344us) server_negotiation.cc:299] Negotiation successful
0619 16:43:58.502272 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260619 16:43:58.517872 7789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:58.485341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55130 (local address 127.0.97.60:35535)
0619 16:43:58.485545 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0619 16:43:58.485551 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:43:58.502717 (+ 17166us) server_negotiation.cc:407] Connection header received
0619 16:43:58.502961 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:43:58.502968 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:43:58.503050 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:43:58.503118 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:43:58.504987 (+ 1869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:58.506142 (+ 1155us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:58.507952 (+ 1810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:58.508621 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:58.508755 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:43:58.511789 (+ 3034us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:43:58.511796 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:43:58.511801 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:43:58.511837 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:43:58.515196 (+ 3359us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:43:58.515902 (+ 706us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:43:58.515908 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:43:58.515912 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:43:58.516047 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:43:58.516432 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:43:58.516437 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:43:58.516440 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:43:58.516761 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:43:58.516989 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0619 16:43:58.517290 (+ 301us) server_negotiation.cc:299] Negotiation successful
0619 16:43:58.517485 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260619 16:43:58.607985 6512 catalog_manager.cc:4841] TS 449f7e56e3754b22a114d8f19ed82f76 (127.0.97.2:34331): tablet aec7d89f1a054af29ed83f30837f7c41 (table test-table [id=6e015cbc439c499bb035b38a4d48fc44]) successfully deleted
W20260619 16:43:58.608650 6512 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.0.97.2:34331, user_credentials={real_user=slave}} blocked reactor thread for 117658us
I20260619 16:43:58.610023 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu with pid 6495
I20260619 16:43:58.645972 7790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:58.630479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42166 (local address 127.0.97.61:42337)
0619 16:43:58.630683 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0619 16:43:58.630703 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0619 16:43:58.631282 (+ 579us) server_negotiation.cc:407] Connection header received
0619 16:43:58.631579 (+ 297us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:43:58.631585 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:43:58.631666 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:43:58.631749 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:43:58.634125 (+ 2376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:58.634914 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:58.636615 (+ 1701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:58.637239 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:58.637357 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:43:58.639926 (+ 2569us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:43:58.639931 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:43:58.639934 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:43:58.639967 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:43:58.642935 (+ 2968us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:43:58.643705 (+ 770us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:43:58.643710 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:43:58.643713 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:43:58.643862 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:43:58.644213 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:43:58.644226 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:43:58.644230 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:43:58.644544 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:43:58.644770 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0619 16:43:58.645399 (+ 629us) server_negotiation.cc:299] Negotiation successful
0619 16:43:58.645614 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260619 16:43:58.646492 7789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:58.631922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55138 (local address 127.0.97.60:35535)
0619 16:43:58.632109 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0619 16:43:58.632115 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:43:58.632348 (+ 233us) server_negotiation.cc:407] Connection header received
0619 16:43:58.632509 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:43:58.632514 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:43:58.632580 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:43:58.632658 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:43:58.634277 (+ 1619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:58.635357 (+ 1080us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:58.636957 (+ 1600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:58.637421 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:58.637534 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:43:58.640585 (+ 3051us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:43:58.640592 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:43:58.640597 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:43:58.640638 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:43:58.643735 (+ 3097us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:43:58.644431 (+ 696us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:43:58.644436 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:43:58.644440 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:43:58.644555 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:43:58.644909 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:43:58.644913 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:43:58.644915 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:43:58.645162 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:43:58.645372 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0619 16:43:58.645661 (+ 289us) server_negotiation.cc:299] Negotiation successful
0619 16:43:58.646193 (+ 532us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260619 16:43:59.428591 7852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:59.427926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46817 (local address 127.0.97.1:52019)
0619 16:43:59.428338 (+ 412us) negotiation.cc:107] Waiting for socket to connect
0619 16:43:59.428446 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":303,"thread_start_us":207,"threads_started":1}
W20260619 16:43:59.428906 7494 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:46817 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.97.62:46817: connect: Connection refused (error 111)
I20260619 16:43:59.435204 7853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:59.432596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46817 (local address 127.0.97.2:51507)
0619 16:43:59.434874 (+ 2278us) negotiation.cc:107] Waiting for socket to connect
0619 16:43:59.435009 (+ 135us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":2168,"thread_start_us":2061,"threads_started":1}
W20260619 16:43:59.435717 7637 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:46817 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.97.62:46817: connect: Connection refused (error 111)
I20260619 16:43:59.460907 7854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:59.460253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46817 (local address 127.0.97.3:43795)
0619 16:43:59.460619 (+ 366us) negotiation.cc:107] Waiting for socket to connect
0619 16:43:59.460723 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":190,"threads_started":1}
W20260619 16:43:59.461261 7786 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:46817 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.97.62:46817: connect: Connection refused (error 111)
I20260619 16:43:59.957527 7855 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2e547db70ef542a88009307c8181f39d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5ba5d12f1093420e96661d1c478add9b)
I20260619 16:43:59.957870 7855 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2e547db70ef542a88009307c8181f39d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "5ba5d12f1093420e96661d1c478add9b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46817 } } peers { permanent_uuid: "85d1444e474f4e189618fb22f5a42908" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42337 } attrs { promote: false } } peers { permanent_uuid: "2e547db70ef542a88009307c8181f39d" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35535 } attrs { promote: false } }
I20260619 16:43:59.958178 7856 raft_consensus.cc:482] T 00000000000000000000000000000000 P 85d1444e474f4e189618fb22f5a42908 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5ba5d12f1093420e96661d1c478add9b)
I20260619 16:43:59.958396 7856 raft_consensus.cc:504] T 00000000000000000000000000000000 P 85d1444e474f4e189618fb22f5a42908 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "5ba5d12f1093420e96661d1c478add9b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46817 } } peers { permanent_uuid: "85d1444e474f4e189618fb22f5a42908" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42337 } attrs { promote: false } } peers { permanent_uuid: "2e547db70ef542a88009307c8181f39d" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35535 } attrs { promote: false } }
I20260619 16:43:59.959106 7856 leader_election.cc:288] T 00000000000000000000000000000000 P 85d1444e474f4e189618fb22f5a42908 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5ba5d12f1093420e96661d1c478add9b (127.0.97.62:46817), 2e547db70ef542a88009307c8181f39d (127.0.97.60:35535)
I20260619 16:43:59.959370 7855 leader_election.cc:288] T 00000000000000000000000000000000 P 2e547db70ef542a88009307c8181f39d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5ba5d12f1093420e96661d1c478add9b (127.0.97.62:46817), 85d1444e474f4e189618fb22f5a42908 (127.0.97.61:42337)
I20260619 16:43:59.960496 7857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:59.959425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46817 (local address 127.0.0.1:46546)
0619 16:43:59.960011 (+ 586us) negotiation.cc:107] Waiting for socket to connect
0619 16:43:59.960178 (+ 167us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":463,"thread_start_us":327,"threads_started":1}
W20260619 16:43:59.961516 7111 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46817: connect: Connection refused (error 111)
I20260619 16:43:59.962520 7859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:59.959452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46817 (local address 127.0.0.1:46552)
0619 16:43:59.962134 (+ 2682us) negotiation.cc:107] Waiting for socket to connect
0619 16:43:59.962286 (+ 152us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":2543,"threads_started":1}
I20260619 16:43:59.971710 7860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:59.959726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42174 (local address 127.0.97.61:42337)
0619 16:43:59.961649 (+ 1923us) server_negotiation.cc:206] Beginning negotiation
0619 16:43:59.961655 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:43:59.962955 (+ 1300us) server_negotiation.cc:407] Connection header received
0619 16:43:59.963191 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:43:59.963197 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:43:59.963291 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:43:59.963358 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:43:59.965056 (+ 1698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:59.966181 (+ 1125us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:59.968918 (+ 2737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:59.970349 (+ 1431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:59.970452 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:43:59.970564 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0619 16:43:59.971165 (+ 601us) server_negotiation.cc:299] Negotiation successful
0619 16:43:59.971320 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1797,"thread_start_us":1628,"threads_started":1}
I20260619 16:43:59.972863 7859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:59.959533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:42337 (local address 127.0.0.1:42174)
0619 16:43:59.962861 (+ 3328us) negotiation.cc:107] Waiting for socket to connect
0619 16:43:59.962879 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 16:43:59.963051 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:43:59.963628 (+ 577us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:43:59.963633 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:43:59.963680 (+ 47us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:43:59.964888 (+ 1208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:43:59.964903 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:43:59.966362 (+ 1459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:43:59.966367 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:43:59.968696 (+ 2329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:43:59.968714 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:43:59.970757 (+ 2043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:43:59.970764 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:43:59.970978 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:43:59.971014 (+ 36us) client_negotiation.cc:769] Sending connection context
0619 16:43:59.972095 (+ 1081us) client_negotiation.cc:240] Negotiation successful
0619 16:43:59.972269 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3192,"thread_start_us":2334,"threads_started":1}
I20260619 16:43:59.974014 7861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:59.973375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46817 (local address 127.0.0.1:46566)
0619 16:43:59.973561 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0619 16:43:59.973762 (+ 201us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260619 16:43:59.973904 7301 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46817: connect: Connection refused (error 111)
I20260619 16:43:59.974109 7133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2e547db70ef542a88009307c8181f39d" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "85d1444e474f4e189618fb22f5a42908" is_pre_election: true
W20260619 16:43:59.974328 7111 leader_election.cc:334] T 00000000000000000000000000000000 P 85d1444e474f4e189618fb22f5a42908 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 5ba5d12f1093420e96661d1c478add9b (127.0.97.62:46817): Network error: Client connection negotiation failed: client connection to 127.0.97.62:46817: connect: Connection refused (error 111)
I20260619 16:43:59.974493 7133 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 85d1444e474f4e189618fb22f5a42908 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2e547db70ef542a88009307c8181f39d in term 1.
I20260619 16:43:59.974987 7299 leader_election.cc:302] T 00000000000000000000000000000000 P 2e547db70ef542a88009307c8181f39d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2e547db70ef542a88009307c8181f39d, 85d1444e474f4e189618fb22f5a42908; no voters:
I20260619 16:43:59.975381 7855 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2e547db70ef542a88009307c8181f39d [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260619 16:43:59.975502 7855 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2e547db70ef542a88009307c8181f39d [term 1 FOLLOWER]: Starting leader election (detected failure of leader 5ba5d12f1093420e96661d1c478add9b)
I20260619 16:43:59.975602 7855 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2e547db70ef542a88009307c8181f39d [term 1 FOLLOWER]: Advancing to term 2
I20260619 16:43:59.976125 7859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:59.975671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46817 (local address 127.0.0.1:46580)
0619 16:43:59.975856 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0619 16:43:59.975958 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260619 16:43:59.976408 7301 leader_election.cc:334] T 00000000000000000000000000000000 P 2e547db70ef542a88009307c8181f39d [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 5ba5d12f1093420e96661d1c478add9b (127.0.97.62:46817): Network error: Client connection negotiation failed: client connection to 127.0.97.62:46817: connect: Connection refused (error 111)
I20260619 16:43:59.978348 7855 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2e547db70ef542a88009307c8181f39d [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "5ba5d12f1093420e96661d1c478add9b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46817 } } peers { permanent_uuid: "85d1444e474f4e189618fb22f5a42908" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42337 } attrs { promote: false } } peers { permanent_uuid: "2e547db70ef542a88009307c8181f39d" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35535 } attrs { promote: false } }
I20260619 16:43:59.979072 7855 leader_election.cc:288] T 00000000000000000000000000000000 P 2e547db70ef542a88009307c8181f39d [CANDIDATE]: Term 2 election: Requested vote from peers 5ba5d12f1093420e96661d1c478add9b (127.0.97.62:46817), 85d1444e474f4e189618fb22f5a42908 (127.0.97.61:42337)
I20260619 16:43:59.979454 7133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2e547db70ef542a88009307c8181f39d" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "85d1444e474f4e189618fb22f5a42908"
I20260619 16:43:59.979676 7133 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 85d1444e474f4e189618fb22f5a42908 [term 1 FOLLOWER]: Advancing to term 2
I20260619 16:43:59.979836 7857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:59.960077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:35535 (local address 127.0.0.1:55142)
0619 16:43:59.960814 (+ 737us) negotiation.cc:107] Waiting for socket to connect
0619 16:43:59.960834 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 16:43:59.960970 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:43:59.973377 (+ 12407us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:43:59.973382 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:43:59.973411 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:43:59.973756 (+ 345us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:43:59.973769 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:43:59.975483 (+ 1714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:43:59.975488 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:43:59.977344 (+ 1856us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:43:59.977356 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:43:59.978941 (+ 1585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:43:59.978947 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:43:59.979117 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:43:59.979148 (+ 31us) client_negotiation.cc:769] Sending connection context
0619 16:43:59.979238 (+ 90us) client_negotiation.cc:240] Negotiation successful
0619 16:43:59.979390 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":629,"thread_start_us":1141,"threads_started":1}
I20260619 16:43:59.979962 7863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:59.962111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55142 (local address 127.0.97.60:35535)
0619 16:43:59.973020 (+ 10909us) server_negotiation.cc:206] Beginning negotiation
0619 16:43:59.973026 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:43:59.973046 (+ 20us) server_negotiation.cc:407] Connection header received
0619 16:43:59.973104 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:43:59.973109 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:43:59.973194 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:43:59.973893 (+ 699us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:43:59.974165 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:59.975359 (+ 1194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:59.977489 (+ 2130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:43:59.978710 (+ 1221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:43:59.978898 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:43:59.979003 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0619 16:43:59.979519 (+ 516us) server_negotiation.cc:299] Negotiation successful
0619 16:43:59.979648 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10777,"thread_start_us":10301,"threads_started":1}
I20260619 16:43:59.980544 7325 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "85d1444e474f4e189618fb22f5a42908" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "2e547db70ef542a88009307c8181f39d" is_pre_election: true
I20260619 16:43:59.980881 7325 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 2e547db70ef542a88009307c8181f39d [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 85d1444e474f4e189618fb22f5a42908 in current term 2: Already voted for candidate 2e547db70ef542a88009307c8181f39d in this term.
I20260619 16:43:59.981379 7111 leader_election.cc:302] T 00000000000000000000000000000000 P 85d1444e474f4e189618fb22f5a42908 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85d1444e474f4e189618fb22f5a42908; no voters: 2e547db70ef542a88009307c8181f39d, 5ba5d12f1093420e96661d1c478add9b
I20260619 16:43:59.982172 7133 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 85d1444e474f4e189618fb22f5a42908 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2e547db70ef542a88009307c8181f39d in term 2.
I20260619 16:43:59.982321 7856 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 85d1444e474f4e189618fb22f5a42908 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 16:43:59.982542 7299 leader_election.cc:302] T 00000000000000000000000000000000 P 2e547db70ef542a88009307c8181f39d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2e547db70ef542a88009307c8181f39d, 85d1444e474f4e189618fb22f5a42908; no voters:
I20260619 16:43:59.982792 7855 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2e547db70ef542a88009307c8181f39d [term 2 FOLLOWER]: Leader election won for term 2
I20260619 16:43:59.983228 7855 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2e547db70ef542a88009307c8181f39d [term 2 LEADER]: Becoming Leader. State: Replica: 2e547db70ef542a88009307c8181f39d, State: Running, Role: LEADER
I20260619 16:43:59.983598 7859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:59.983243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46817 (local address 127.0.0.1:46596)
0619 16:43:59.983426 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0619 16:43:59.983493 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260619 16:43:59.983618 7855 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2e547db70ef542a88009307c8181f39d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "5ba5d12f1093420e96661d1c478add9b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46817 } } peers { permanent_uuid: "85d1444e474f4e189618fb22f5a42908" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42337 } attrs { promote: false } } peers { permanent_uuid: "2e547db70ef542a88009307c8181f39d" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35535 } attrs { promote: false } }
I20260619 16:43:59.984452 7859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:43:59.984106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46817 (local address 127.0.0.1:46598)
0619 16:43:59.984246 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0619 16:43:59.984329 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260619 16:43:59.984645 7301 leader_election.cc:334] T 00000000000000000000000000000000 P 2e547db70ef542a88009307c8181f39d [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 5ba5d12f1093420e96661d1c478add9b (127.0.97.62:46817): Network error: Client connection negotiation failed: client connection to 127.0.97.62:46817: connect: Connection refused (error 111)
I20260619 16:43:59.985695 7865 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2e547db70ef542a88009307c8181f39d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2e547db70ef542a88009307c8181f39d. Latest consensus state: current_term: 2 leader_uuid: "2e547db70ef542a88009307c8181f39d" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "5ba5d12f1093420e96661d1c478add9b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46817 } } peers { permanent_uuid: "85d1444e474f4e189618fb22f5a42908" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42337 } attrs { promote: false } } peers { permanent_uuid: "2e547db70ef542a88009307c8181f39d" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35535 } attrs { promote: false } } }
I20260619 16:43:59.986032 7865 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2e547db70ef542a88009307c8181f39d [sys.catalog]: This master's current role is: LEADER
I20260619 16:43:59.986565 7867 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:43:59.989825 7867 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:43:59.990562 7867 catalog_manager.cc:1259] Loaded cluster ID: dc389b63eb214c9bb142cff4aef89cd5
I20260619 16:43:59.990626 7867 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:43:59.991663 7867 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:43:59.992383 7867 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 2e547db70ef542a88009307c8181f39d: Loaded TSK: 0
I20260619 16:43:59.993147 7867 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 16:44:00.025678 7314 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55138:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 16:44:00.032577 7314 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 16:44:00.036939 7859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:44:00.036518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46817 (local address 127.0.0.1:46614)
0619 16:44:00.036672 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0619 16:44:00.036767 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260619 16:44:00.037062 7133 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 85d1444e474f4e189618fb22f5a42908 [term 2 FOLLOWER]: Refusing update from remote peer 2e547db70ef542a88009307c8181f39d: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260619 16:44:00.037890 7865 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 2e547db70ef542a88009307c8181f39d [LEADER]: Connected to new peer: Peer: permanent_uuid: "85d1444e474f4e189618fb22f5a42908" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42337 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260619 16:44:00.038044 7859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:44:00.037607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46817 (local address 127.0.0.1:46618)
0619 16:44:00.037816 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0619 16:44:00.037913 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260619 16:44:00.038434 7301 consensus_peers.cc:487] T 00000000000000000000000000000000 P 2e547db70ef542a88009307c8181f39d -> Peer 5ba5d12f1093420e96661d1c478add9b (127.0.97.62:46817): Couldn't send request to peer 5ba5d12f1093420e96661d1c478add9b. Status: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46817: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260619 16:44:00.040764 7856 sys_catalog.cc:455] T 00000000000000000000000000000000 P 85d1444e474f4e189618fb22f5a42908 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2e547db70ef542a88009307c8181f39d. Latest consensus state: current_term: 2 leader_uuid: "2e547db70ef542a88009307c8181f39d" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "5ba5d12f1093420e96661d1c478add9b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46817 } } peers { permanent_uuid: "85d1444e474f4e189618fb22f5a42908" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42337 } attrs { promote: false } } peers { permanent_uuid: "2e547db70ef542a88009307c8181f39d" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35535 } attrs { promote: false } } }
I20260619 16:44:00.041000 7856 sys_catalog.cc:458] T 00000000000000000000000000000000 P 85d1444e474f4e189618fb22f5a42908 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 16:44:00.042387 7855 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2e547db70ef542a88009307c8181f39d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "2e547db70ef542a88009307c8181f39d" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "5ba5d12f1093420e96661d1c478add9b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46817 } } peers { permanent_uuid: "85d1444e474f4e189618fb22f5a42908" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42337 } attrs { promote: false } } peers { permanent_uuid: "2e547db70ef542a88009307c8181f39d" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35535 } attrs { promote: false } } }
I20260619 16:44:00.042694 7855 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2e547db70ef542a88009307c8181f39d [sys.catalog]: This master's current role is: LEADER
I20260619 16:44:00.043735 7856 sys_catalog.cc:455] T 00000000000000000000000000000000 P 85d1444e474f4e189618fb22f5a42908 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "2e547db70ef542a88009307c8181f39d" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "5ba5d12f1093420e96661d1c478add9b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46817 } } peers { permanent_uuid: "85d1444e474f4e189618fb22f5a42908" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42337 } attrs { promote: false } } peers { permanent_uuid: "2e547db70ef542a88009307c8181f39d" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35535 } attrs { promote: false } } }
I20260619 16:44:00.044561 7856 sys_catalog.cc:458] T 00000000000000000000000000000000 P 85d1444e474f4e189618fb22f5a42908 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 16:44:00.045097 7869 mvcc.cc:205] Tried to move back new op lower bound from 7298610954384932864 to 7298610954176438272. Current Snapshot: MvccSnapshot[applied={T|T < 7298610954384932864}]
I20260619 16:44:00.066556 7859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:44:00.050681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:40321 (local address 127.0.0.1:33770)
0619 16:44:00.050887 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0619 16:44:00.050906 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 16:44:00.051012 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:44:00.061069 (+ 10057us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:44:00.061074 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:44:00.061100 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:44:00.061278 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:44:00.061289 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:44:00.062576 (+ 1287us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:44:00.062582 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:44:00.064239 (+ 1657us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:44:00.064250 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:44:00.065790 (+ 1540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:44:00.065796 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:44:00.066084 (+ 288us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:44:00.066118 (+ 34us) client_negotiation.cc:769] Sending connection context
0619 16:44:00.066197 (+ 79us) client_negotiation.cc:240] Negotiation successful
0619 16:44:00.066305 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":87,"mutex_wait_us":128}
I20260619 16:44:00.066636 7874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:44:00.050821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33770 (local address 127.0.97.1:40321)
0619 16:44:00.060793 (+ 9972us) server_negotiation.cc:206] Beginning negotiation
0619 16:44:00.060798 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:44:00.060814 (+ 16us) server_negotiation.cc:407] Connection header received
0619 16:44:00.060871 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:44:00.060874 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:44:00.060929 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:44:00.061023 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:44:00.061443 (+ 420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:44:00.062390 (+ 947us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:44:00.064419 (+ 2029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:44:00.065581 (+ 1162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:44:00.065939 (+ 358us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:44:00.066099 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0619 16:44:00.066232 (+ 133us) server_negotiation.cc:299] Negotiation successful
0619 16:44:00.066370 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9869,"thread_start_us":10204,"threads_started":1}
I20260619 16:44:00.067565 7428 tablet_service.cc:1467] Processing CreateTablet for tablet cbf4f2c9df3d4491b7ca54c1d065ff7f (DEFAULT_TABLE table=test-table [id=0688b616a16c4e5badd8cd8b009d8273]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:44:00.067780 7876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:44:00.052220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:34331 (local address 127.0.0.1:46548)
0619 16:44:00.052690 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0619 16:44:00.052706 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 16:44:00.052834 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:44:00.061194 (+ 8360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:44:00.061199 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:44:00.061224 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:44:00.061565 (+ 341us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:44:00.061574 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:44:00.062949 (+ 1375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:44:00.062955 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:44:00.064843 (+ 1888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:44:00.064855 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:44:00.067025 (+ 2170us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:44:00.067031 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:44:00.067195 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:44:00.067229 (+ 34us) client_negotiation.cc:769] Sending connection context
0619 16:44:00.067324 (+ 95us) client_negotiation.cc:240] Negotiation successful
0619 16:44:00.067451 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":360,"thread_start_us":404,"threads_started":1}
I20260619 16:44:00.068048 7428 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cbf4f2c9df3d4491b7ca54c1d065ff7f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:44:00.068593 7877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:44:00.052502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46548 (local address 127.0.97.2:34331)
0619 16:44:00.060710 (+ 8208us) server_negotiation.cc:206] Beginning negotiation
0619 16:44:00.060718 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 16:44:00.060743 (+ 25us) server_negotiation.cc:407] Connection header received
0619 16:44:00.060826 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:44:00.060831 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:44:00.060913 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:44:00.061186 (+ 273us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:44:00.061729 (+ 543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:44:00.062802 (+ 1073us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:44:00.065013 (+ 2211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:44:00.065839 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:44:00.066002 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:44:00.066104 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0619 16:44:00.068153 (+ 2049us) server_negotiation.cc:299] Negotiation successful
0619 16:44:00.068315 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8060,"thread_start_us":7938,"threads_started":1}
I20260619 16:44:00.068652 7858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:44:00.051026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:40501 (local address 127.0.0.1:51236)
0619 16:44:00.051217 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0619 16:44:00.051233 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 16:44:00.051324 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:44:00.062422 (+ 11098us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:44:00.062427 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:44:00.062453 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:44:00.062882 (+ 429us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:44:00.062895 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:44:00.064365 (+ 1470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:44:00.064369 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:44:00.066411 (+ 2042us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:44:00.066423 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:44:00.067986 (+ 1563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:44:00.067993 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:44:00.068169 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:44:00.068199 (+ 30us) client_negotiation.cc:769] Sending connection context
0619 16:44:00.068254 (+ 55us) client_negotiation.cc:240] Negotiation successful
0619 16:44:00.068380 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":80,"mutex_wait_us":45}
I20260619 16:44:00.069315 7571 tablet_service.cc:1467] Processing CreateTablet for tablet cbf4f2c9df3d4491b7ca54c1d065ff7f (DEFAULT_TABLE table=test-table [id=0688b616a16c4e5badd8cd8b009d8273]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:44:00.069435 7875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:44:00.051128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51236 (local address 127.0.97.3:40501)
0619 16:44:00.062087 (+ 10959us) server_negotiation.cc:206] Beginning negotiation
0619 16:44:00.062093 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:44:00.062113 (+ 20us) server_negotiation.cc:407] Connection header received
0619 16:44:00.062175 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:44:00.062180 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:44:00.062271 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:44:00.063037 (+ 766us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:44:00.063211 (+ 174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:44:00.064224 (+ 1013us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:44:00.066582 (+ 2358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:44:00.067766 (+ 1184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:44:00.068915 (+ 1149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:44:00.069001 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0619 16:44:00.069052 (+ 51us) server_negotiation.cc:299] Negotiation successful
0619 16:44:00.069173 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10839,"thread_start_us":10747,"threads_started":1}
I20260619 16:44:00.069698 7571 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cbf4f2c9df3d4491b7ca54c1d065ff7f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:44:00.070091 7720 tablet_service.cc:1467] Processing CreateTablet for tablet cbf4f2c9df3d4491b7ca54c1d065ff7f (DEFAULT_TABLE table=test-table [id=0688b616a16c4e5badd8cd8b009d8273]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:44:00.070452 7720 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cbf4f2c9df3d4491b7ca54c1d065ff7f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:44:00.072872 7878 tablet_bootstrap.cc:492] T cbf4f2c9df3d4491b7ca54c1d065ff7f P 4e87af70e6654dff8d6f88b6b739ee42: Bootstrap starting.
I20260619 16:44:00.074776 7878 tablet_bootstrap.cc:654] T cbf4f2c9df3d4491b7ca54c1d065ff7f P 4e87af70e6654dff8d6f88b6b739ee42: Neither blocks nor log segments found. Creating new log.
I20260619 16:44:00.076642 7878 tablet_bootstrap.cc:492] T cbf4f2c9df3d4491b7ca54c1d065ff7f P 4e87af70e6654dff8d6f88b6b739ee42: No bootstrap required, opened a new log
I20260619 16:44:00.076767 7878 ts_tablet_manager.cc:1397] T cbf4f2c9df3d4491b7ca54c1d065ff7f P 4e87af70e6654dff8d6f88b6b739ee42: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260619 16:44:00.077344 7881 tablet_bootstrap.cc:492] T cbf4f2c9df3d4491b7ca54c1d065ff7f P 399a36acb0b447e8b9c0d6992bc86ea2: Bootstrap starting.
I20260619 16:44:00.077521 7878 raft_consensus.cc:348] T cbf4f2c9df3d4491b7ca54c1d065ff7f P 4e87af70e6654dff8d6f88b6b739ee42 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e87af70e6654dff8d6f88b6b739ee42" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40321 } } peers { permanent_uuid: "399a36acb0b447e8b9c0d6992bc86ea2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40501 } } peers { permanent_uuid: "449f7e56e3754b22a114d8f19ed82f76" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34331 } }
I20260619 16:44:00.077768 7878 raft_consensus.cc:374] T cbf4f2c9df3d4491b7ca54c1d065ff7f P 4e87af70e6654dff8d6f88b6b739ee42 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:44:00.077867 7878 raft_consensus.cc:729] T cbf4f2c9df3d4491b7ca54c1d065ff7f P 4e87af70e6654dff8d6f88b6b739ee42 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e87af70e6654dff8d6f88b6b739ee42, State: Initialized, Role: FOLLOWER
I20260619 16:44:00.078079 7878 consensus_queue.cc:260] T cbf4f2c9df3d4491b7ca54c1d065ff7f P 4e87af70e6654dff8d6f88b6b739ee42 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e87af70e6654dff8d6f88b6b739ee42" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40321 } } peers { permanent_uuid: "399a36acb0b447e8b9c0d6992bc86ea2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40501 } } peers { permanent_uuid: "449f7e56e3754b22a114d8f19ed82f76" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34331 } }
I20260619 16:44:00.078830 7878 ts_tablet_manager.cc:1428] T cbf4f2c9df3d4491b7ca54c1d065ff7f P 4e87af70e6654dff8d6f88b6b739ee42: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260619 16:44:00.079140 7881 tablet_bootstrap.cc:654] T cbf4f2c9df3d4491b7ca54c1d065ff7f P 399a36acb0b447e8b9c0d6992bc86ea2: Neither blocks nor log segments found. Creating new log.
I20260619 16:44:00.076058 7880 tablet_bootstrap.cc:492] T cbf4f2c9df3d4491b7ca54c1d065ff7f P 449f7e56e3754b22a114d8f19ed82f76: Bootstrap starting.
I20260619 16:44:00.080258 7883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:44:00.079500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46817 (local address 127.0.97.1:53779)
0619 16:44:00.079940 (+ 440us) negotiation.cc:107] Waiting for socket to connect
0619 16:44:00.080051 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":323,"threads_started":1}
I20260619 16:44:00.080945 7880 tablet_bootstrap.cc:654] T cbf4f2c9df3d4491b7ca54c1d065ff7f P 449f7e56e3754b22a114d8f19ed82f76: Neither blocks nor log segments found. Creating new log.
I20260619 16:44:00.083455 7493 heartbeater.cc:503] Master 127.0.97.60:35535 was elected leader, sending a full tablet report...
I20260619 16:44:00.083467 7880 tablet_bootstrap.cc:492] T cbf4f2c9df3d4491b7ca54c1d065ff7f P 449f7e56e3754b22a114d8f19ed82f76: No bootstrap required, opened a new log
I20260619 16:44:00.083608 7880 ts_tablet_manager.cc:1397] T cbf4f2c9df3d4491b7ca54c1d065ff7f P 449f7e56e3754b22a114d8f19ed82f76: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.000s
I20260619 16:44:00.084419 7880 raft_consensus.cc:348] T cbf4f2c9df3d4491b7ca54c1d065ff7f P 449f7e56e3754b22a114d8f19ed82f76 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e87af70e6654dff8d6f88b6b739ee42" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40321 } } peers { permanent_uuid: "399a36acb0b447e8b9c0d6992bc86ea2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40501 } } peers { permanent_uuid: "449f7e56e3754b22a114d8f19ed82f76" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34331 } }
I20260619 16:44:00.084678 7880 raft_consensus.cc:374] T cbf4f2c9df3d4491b7ca54c1d065ff7f P 449f7e56e3754b22a114d8f19ed82f76 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:44:00.084815 7880 raft_consensus.cc:729] T cbf4f2c9df3d4491b7ca54c1d065ff7f P 449f7e56e3754b22a114d8f19ed82f76 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 449f7e56e3754b22a114d8f19ed82f76, State: Initialized, Role: FOLLOWER
I20260619 16:44:00.085114 7880 consensus_queue.cc:260] T cbf4f2c9df3d4491b7ca54c1d065ff7f P 449f7e56e3754b22a114d8f19ed82f76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e87af70e6654dff8d6f88b6b739ee42" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40321 } } peers { permanent_uuid: "399a36acb0b447e8b9c0d6992bc86ea2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40501 } } peers { permanent_uuid: "449f7e56e3754b22a114d8f19ed82f76" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34331 } }
I20260619 16:44:00.085963 7880 ts_tablet_manager.cc:1428] T cbf4f2c9df3d4491b7ca54c1d065ff7f P 449f7e56e3754b22a114d8f19ed82f76: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260619 16:44:00.087088 7887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:44:00.086201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46817 (local address 127.0.97.2:58247)
0619 16:44:00.086706 (+ 505us) negotiation.cc:107] Waiting for socket to connect
0619 16:44:00.086836 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":364,"thread_start_us":242,"threads_started":1}
I20260619 16:44:00.087965 7636 heartbeater.cc:503] Master 127.0.97.60:35535 was elected leader, sending a full tablet report...
I20260619 16:44:00.088686 7887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:44:00.088205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46817 (local address 127.0.97.2:39481)
0619 16:44:00.088474 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0619 16:44:00.088553 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":169}
I20260619 16:44:00.089120 7881 tablet_bootstrap.cc:492] T cbf4f2c9df3d4491b7ca54c1d065ff7f P 399a36acb0b447e8b9c0d6992bc86ea2: No bootstrap required, opened a new log
I20260619 16:44:00.089321 7881 ts_tablet_manager.cc:1397] T cbf4f2c9df3d4491b7ca54c1d065ff7f P 399a36acb0b447e8b9c0d6992bc86ea2: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.007s
I20260619 16:44:00.090204 7881 raft_consensus.cc:348] T cbf4f2c9df3d4491b7ca54c1d065ff7f P 399a36acb0b447e8b9c0d6992bc86ea2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e87af70e6654dff8d6f88b6b739ee42" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40321 } } peers { permanent_uuid: "399a36acb0b447e8b9c0d6992bc86ea2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40501 } } peers { permanent_uuid: "449f7e56e3754b22a114d8f19ed82f76" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34331 } }
I20260619 16:44:00.090426 7881 raft_consensus.cc:374] T cbf4f2c9df3d4491b7ca54c1d065ff7f P 399a36acb0b447e8b9c0d6992bc86ea2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:44:00.090521 7881 raft_consensus.cc:729] T cbf4f2c9df3d4491b7ca54c1d065ff7f P 399a36acb0b447e8b9c0d6992bc86ea2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 399a36acb0b447e8b9c0d6992bc86ea2, State: Initialized, Role: FOLLOWER
I20260619 16:44:00.090744 7881 consensus_queue.cc:260] T cbf4f2c9df3d4491b7ca54c1d065ff7f P 399a36acb0b447e8b9c0d6992bc86ea2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e87af70e6654dff8d6f88b6b739ee42" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40321 } } peers { permanent_uuid: "399a36acb0b447e8b9c0d6992bc86ea2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40501 } } peers { permanent_uuid: "449f7e56e3754b22a114d8f19ed82f76" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34331 } }
I20260619 16:44:00.091348 7881 ts_tablet_manager.cc:1428] T cbf4f2c9df3d4491b7ca54c1d065ff7f P 399a36acb0b447e8b9c0d6992bc86ea2: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260619 16:44:00.092519 7889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:44:00.091644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46817 (local address 127.0.97.3:39465)
0619 16:44:00.092130 (+ 486us) negotiation.cc:107] Waiting for socket to connect
0619 16:44:00.092257 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":353,"thread_start_us":227,"threads_started":1}
I20260619 16:44:00.093325 7889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:44:00.093016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46817 (local address 127.0.97.3:51897)
0619 16:44:00.093140 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0619 16:44:00.093212 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260619 16:44:00.093631 7785 heartbeater.cc:503] Master 127.0.97.60:35535 was elected leader, sending a full tablet report...
I20260619 16:44:00.111081 7888 raft_consensus.cc:482] T cbf4f2c9df3d4491b7ca54c1d065ff7f P 399a36acb0b447e8b9c0d6992bc86ea2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 16:44:00.111280 7888 raft_consensus.cc:504] T cbf4f2c9df3d4491b7ca54c1d065ff7f P 399a36acb0b447e8b9c0d6992bc86ea2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e87af70e6654dff8d6f88b6b739ee42" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40321 } } peers { permanent_uuid: "399a36acb0b447e8b9c0d6992bc86ea2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40501 } } peers { permanent_uuid: "449f7e56e3754b22a114d8f19ed82f76" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34331 } }
I20260619 16:44:00.112700 7888 leader_election.cc:288] T cbf4f2c9df3d4491b7ca54c1d065ff7f P 399a36acb0b447e8b9c0d6992bc86ea2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4e87af70e6654dff8d6f88b6b739ee42 (127.0.97.1:40321), 449f7e56e3754b22a114d8f19ed82f76 (127.0.97.2:34331)
I20260619 16:44:00.119393 7877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:44:00.113089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:37367 (local address 127.0.97.2:34331)
0619 16:44:00.113664 (+ 575us) server_negotiation.cc:206] Beginning negotiation
0619 16:44:00.113670 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:44:00.113698 (+ 28us) server_negotiation.cc:407] Connection header received
0619 16:44:00.113799 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:44:00.113804 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:44:00.113880 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:44:00.113984 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:44:00.114547 (+ 563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:44:00.115389 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:44:00.117319 (+ 1930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:44:00.118351 (+ 1032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:44:00.118471 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:44:00.118545 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0619 16:44:00.118975 (+ 430us) server_negotiation.cc:299] Negotiation successful
0619 16:44:00.119105 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":417}
I20260619 16:44:00.119429 7889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:44:00.112958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:34331 (local address 127.0.97.3:37367)
0619 16:44:00.113221 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0619 16:44:00.113241 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 16:44:00.113388 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:44:00.114016 (+ 628us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:44:00.114022 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:44:00.114051 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:44:00.114329 (+ 278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:44:00.114341 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:44:00.115544 (+ 1203us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:44:00.115550 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:44:00.117113 (+ 1563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:44:00.117126 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:44:00.118558 (+ 1432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:44:00.118566 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:44:00.118796 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:44:00.118835 (+ 39us) client_negotiation.cc:769] Sending connection context
0619 16:44:00.118938 (+ 103us) client_negotiation.cc:240] Negotiation successful
0619 16:44:00.119117 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"mutex_wait_us":1,"spinlock_wait_cycles":768}
I20260619 16:44:00.119755 7874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:44:00.113411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:41173 (local address 127.0.97.1:40321)
0619 16:44:00.113762 (+ 351us) server_negotiation.cc:206] Beginning negotiation
0619 16:44:00.113769 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:44:00.114002 (+ 233us) server_negotiation.cc:407] Connection header received
0619 16:44:00.114285 (+ 283us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:44:00.114290 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:44:00.114363 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:44:00.114470 (+ 107us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:44:00.115018 (+ 548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:44:00.115866 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:44:00.117823 (+ 1957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:44:00.118775 (+ 952us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:44:00.118931 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:44:00.119055 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0619 16:44:00.119357 (+ 302us) server_negotiation.cc:299] Negotiation successful
0619 16:44:00.119495 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":174}
I20260619 16:44:00.119819 7890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:44:00.112978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:40321 (local address 127.0.97.3:41173)
0619 16:44:00.113918 (+ 940us) negotiation.cc:107] Waiting for socket to connect
0619 16:44:00.113934 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 16:44:00.114093 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:44:00.114525 (+ 432us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:44:00.114529 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:44:00.114552 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:44:00.114851 (+ 299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:44:00.114862 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:44:00.116032 (+ 1170us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:44:00.116036 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:44:00.117613 (+ 1577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:44:00.117627 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:44:00.118985 (+ 1358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:44:00.118991 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:44:00.119186 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:44:00.119220 (+ 34us) client_negotiation.cc:769] Sending connection context
0619 16:44:00.119315 (+ 95us) client_negotiation.cc:240] Negotiation successful
0619 16:44:00.119439 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":834,"mutex_wait_us":71,"thread_start_us":661,"threads_started":1}
I20260619 16:44:00.120497 7591 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cbf4f2c9df3d4491b7ca54c1d065ff7f" candidate_uuid: "399a36acb0b447e8b9c0d6992bc86ea2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "449f7e56e3754b22a114d8f19ed82f76" is_pre_election: true
I20260619 16:44:00.120633 7448 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cbf4f2c9df3d4491b7ca54c1d065ff7f" candidate_uuid: "399a36acb0b447e8b9c0d6992bc86ea2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e87af70e6654dff8d6f88b6b739ee42" is_pre_election: true
I20260619 16:44:00.120838 7591 raft_consensus.cc:2454] T cbf4f2c9df3d4491b7ca54c1d065ff7f P 449f7e56e3754b22a114d8f19ed82f76 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 399a36acb0b447e8b9c0d6992bc86ea2 in term 0.
I20260619 16:44:00.121025 7448 raft_consensus.cc:2454] T cbf4f2c9df3d4491b7ca54c1d065ff7f P 4e87af70e6654dff8d6f88b6b739ee42 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 399a36acb0b447e8b9c0d6992bc86ea2 in term 0.
I20260619 16:44:00.121423 7674 leader_election.cc:302] T cbf4f2c9df3d4491b7ca54c1d065ff7f P 399a36acb0b447e8b9c0d6992bc86ea2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 399a36acb0b447e8b9c0d6992bc86ea2, 449f7e56e3754b22a114d8f19ed82f76; no voters:
I20260619 16:44:00.121826 7888 raft_consensus.cc:2781] T cbf4f2c9df3d4491b7ca54c1d065ff7f P 399a36acb0b447e8b9c0d6992bc86ea2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 16:44:00.121948 7888 raft_consensus.cc:482] T cbf4f2c9df3d4491b7ca54c1d065ff7f P 399a36acb0b447e8b9c0d6992bc86ea2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 16:44:00.122040 7888 raft_consensus.cc:3037] T cbf4f2c9df3d4491b7ca54c1d065ff7f P 399a36acb0b447e8b9c0d6992bc86ea2 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:44:00.124085 7888 raft_consensus.cc:504] T cbf4f2c9df3d4491b7ca54c1d065ff7f P 399a36acb0b447e8b9c0d6992bc86ea2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e87af70e6654dff8d6f88b6b739ee42" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40321 } } peers { permanent_uuid: "399a36acb0b447e8b9c0d6992bc86ea2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40501 } } peers { permanent_uuid: "449f7e56e3754b22a114d8f19ed82f76" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34331 } }
I20260619 16:44:00.124605 7888 leader_election.cc:288] T cbf4f2c9df3d4491b7ca54c1d065ff7f P 399a36acb0b447e8b9c0d6992bc86ea2 [CANDIDATE]: Term 1 election: Requested vote from peers 4e87af70e6654dff8d6f88b6b739ee42 (127.0.97.1:40321), 449f7e56e3754b22a114d8f19ed82f76 (127.0.97.2:34331)
I20260619 16:44:00.125110 7448 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cbf4f2c9df3d4491b7ca54c1d065ff7f" candidate_uuid: "399a36acb0b447e8b9c0d6992bc86ea2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e87af70e6654dff8d6f88b6b739ee42"
I20260619 16:44:00.125116 7591 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cbf4f2c9df3d4491b7ca54c1d065ff7f" candidate_uuid: "399a36acb0b447e8b9c0d6992bc86ea2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "449f7e56e3754b22a114d8f19ed82f76"
I20260619 16:44:00.125267 7448 raft_consensus.cc:3037] T cbf4f2c9df3d4491b7ca54c1d065ff7f P 4e87af70e6654dff8d6f88b6b739ee42 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:44:00.125288 7591 raft_consensus.cc:3037] T cbf4f2c9df3d4491b7ca54c1d065ff7f P 449f7e56e3754b22a114d8f19ed82f76 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:44:00.127187 7591 raft_consensus.cc:2454] T cbf4f2c9df3d4491b7ca54c1d065ff7f P 449f7e56e3754b22a114d8f19ed82f76 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 399a36acb0b447e8b9c0d6992bc86ea2 in term 1.
I20260619 16:44:00.127239 7448 raft_consensus.cc:2454] T cbf4f2c9df3d4491b7ca54c1d065ff7f P 4e87af70e6654dff8d6f88b6b739ee42 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 399a36acb0b447e8b9c0d6992bc86ea2 in term 1.
I20260619 16:44:00.127570 7674 leader_election.cc:302] T cbf4f2c9df3d4491b7ca54c1d065ff7f P 399a36acb0b447e8b9c0d6992bc86ea2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 399a36acb0b447e8b9c0d6992bc86ea2, 449f7e56e3754b22a114d8f19ed82f76; no voters:
I20260619 16:44:00.127826 7888 raft_consensus.cc:2781] T cbf4f2c9df3d4491b7ca54c1d065ff7f P 399a36acb0b447e8b9c0d6992bc86ea2 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:44:00.128249 7888 raft_consensus.cc:686] T cbf4f2c9df3d4491b7ca54c1d065ff7f P 399a36acb0b447e8b9c0d6992bc86ea2 [term 1 LEADER]: Becoming Leader. State: Replica: 399a36acb0b447e8b9c0d6992bc86ea2, State: Running, Role: LEADER
I20260619 16:44:00.128614 7888 consensus_queue.cc:237] T cbf4f2c9df3d4491b7ca54c1d065ff7f P 399a36acb0b447e8b9c0d6992bc86ea2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e87af70e6654dff8d6f88b6b739ee42" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40321 } } peers { permanent_uuid: "399a36acb0b447e8b9c0d6992bc86ea2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40501 } } peers { permanent_uuid: "449f7e56e3754b22a114d8f19ed82f76" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34331 } }
I20260619 16:44:00.129218 7890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:44:00.128793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46817 (local address 127.0.97.3:40387)
0619 16:44:00.128947 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0619 16:44:00.129049 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260619 16:44:00.132121 7313 catalog_manager.cc:5495] T cbf4f2c9df3d4491b7ca54c1d065ff7f P 399a36acb0b447e8b9c0d6992bc86ea2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 399a36acb0b447e8b9c0d6992bc86ea2 (127.0.97.3). New cstate: current_term: 1 leader_uuid: "399a36acb0b447e8b9c0d6992bc86ea2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e87af70e6654dff8d6f88b6b739ee42" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40321 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "399a36acb0b447e8b9c0d6992bc86ea2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40501 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "449f7e56e3754b22a114d8f19ed82f76" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34331 } health_report { overall_health: UNKNOWN } } }
I20260619 16:44:00.160825 7875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:44:00.155754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51250 (local address 127.0.97.3:40501)
0619 16:44:00.155981 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0619 16:44:00.155986 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:44:00.156007 (+ 21us) server_negotiation.cc:407] Connection header received
0619 16:44:00.156201 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:44:00.156205 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:44:00.156291 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:44:00.156429 (+ 138us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 16:44:00.156887 (+ 458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:44:00.157581 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:44:00.158903 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:44:00.159302 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:44:00.159388 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:44:00.159761 (+ 373us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 16:44:00.160269 (+ 508us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 16:44:00.160393 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0619 16:44:00.160512 (+ 119us) server_negotiation.cc:299] Negotiation successful
0619 16:44:00.160575 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260619 16:44:00.164703 7591 raft_consensus.cc:1261] T cbf4f2c9df3d4491b7ca54c1d065ff7f P 449f7e56e3754b22a114d8f19ed82f76 [term 1 FOLLOWER]: Refusing update from remote peer 399a36acb0b447e8b9c0d6992bc86ea2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:44:00.165043 7448 raft_consensus.cc:1261] T cbf4f2c9df3d4491b7ca54c1d065ff7f P 4e87af70e6654dff8d6f88b6b739ee42 [term 1 FOLLOWER]: Refusing update from remote peer 399a36acb0b447e8b9c0d6992bc86ea2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:44:00.165396 7891 consensus_queue.cc:1035] T cbf4f2c9df3d4491b7ca54c1d065ff7f P 399a36acb0b447e8b9c0d6992bc86ea2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "449f7e56e3754b22a114d8f19ed82f76" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34331 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:44:00.165795 7888 consensus_queue.cc:1035] T cbf4f2c9df3d4491b7ca54c1d065ff7f P 399a36acb0b447e8b9c0d6992bc86ea2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e87af70e6654dff8d6f88b6b739ee42" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40321 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:44:00.168423 7883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:44:00.167972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46817 (local address 127.0.97.1:60145)
0619 16:44:00.168107 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0619 16:44:00.168212 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260619 16:44:00.169273 7887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:44:00.168766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46817 (local address 127.0.97.2:47665)
0619 16:44:00.168940 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0619 16:44:00.169070 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260619 16:44:00.174203 7887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:44:00.173655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46817 (local address 127.0.97.2:36519)
0619 16:44:00.173911 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0619 16:44:00.174032 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137}
I20260619 16:44:00.181592 7883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:44:00.181041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46817 (local address 127.0.97.1:52609)
0619 16:44:00.181278 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0619 16:44:00.181370 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113}
I20260619 16:44:00.189613 7890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:44:00.189125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46817 (local address 127.0.97.3:48707)
0619 16:44:00.189361 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0619 16:44:00.189447 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":138}
I20260619 16:44:00.190631 7890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:44:00.190249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46817 (local address 127.0.97.3:51797)
0619 16:44:00.190394 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0619 16:44:00.190476 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260619 16:44:00.197232 7313 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55138:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:44:00.197484 7313 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55138:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:44:00.202813 7313 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2e547db70ef542a88009307c8181f39d: Sending DeleteTablet for 3 replicas of tablet cbf4f2c9df3d4491b7ca54c1d065ff7f
I20260619 16:44:00.203704 7428 tablet_service.cc:1514] Processing DeleteTablet for tablet cbf4f2c9df3d4491b7ca54c1d065ff7f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:44:00 UTC) from {username='oryx', principal='oryx/127.0.97.60@KRBTEST.COM'} at 127.0.0.1:33770
I20260619 16:44:00.204031 7720 tablet_service.cc:1514] Processing DeleteTablet for tablet cbf4f2c9df3d4491b7ca54c1d065ff7f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:44:00 UTC) from {username='oryx', principal='oryx/127.0.97.60@KRBTEST.COM'} at 127.0.0.1:51236
I20260619 16:44:00.204624 7903 tablet_replica.cc:330] T cbf4f2c9df3d4491b7ca54c1d065ff7f P 4e87af70e6654dff8d6f88b6b739ee42: stopping tablet replica
I20260619 16:44:00.204560 7571 tablet_service.cc:1514] Processing DeleteTablet for tablet cbf4f2c9df3d4491b7ca54c1d065ff7f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:44:00 UTC) from {username='oryx', principal='oryx/127.0.97.60@KRBTEST.COM'} at 127.0.0.1:46548
I20260619 16:44:00.204883 7903 raft_consensus.cc:2229] T cbf4f2c9df3d4491b7ca54c1d065ff7f P 4e87af70e6654dff8d6f88b6b739ee42 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 16:44:00.205210 7903 raft_consensus.cc:2258] T cbf4f2c9df3d4491b7ca54c1d065ff7f P 4e87af70e6654dff8d6f88b6b739ee42 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:44:00.205626 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu pid 7319
I20260619 16:44:00.214291 7903 ts_tablet_manager.cc:1905] T cbf4f2c9df3d4491b7ca54c1d065ff7f P 4e87af70e6654dff8d6f88b6b739ee42: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:44:00.214524 7905 tablet_replica.cc:330] T cbf4f2c9df3d4491b7ca54c1d065ff7f P 449f7e56e3754b22a114d8f19ed82f76: stopping tablet replica
I20260619 16:44:00.214821 7905 raft_consensus.cc:2229] T cbf4f2c9df3d4491b7ca54c1d065ff7f P 449f7e56e3754b22a114d8f19ed82f76 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 16:44:00.215170 7905 raft_consensus.cc:2258] T cbf4f2c9df3d4491b7ca54c1d065ff7f P 449f7e56e3754b22a114d8f19ed82f76 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:44:00.215646 7904 tablet_replica.cc:330] T cbf4f2c9df3d4491b7ca54c1d065ff7f P 399a36acb0b447e8b9c0d6992bc86ea2: stopping tablet replica
I20260619 16:44:00.215842 7904 raft_consensus.cc:2229] T cbf4f2c9df3d4491b7ca54c1d065ff7f P 399a36acb0b447e8b9c0d6992bc86ea2 [term 1 LEADER]: Raft consensus shutting down.
I20260619 16:44:00.216437 7904 raft_consensus.cc:2258] T cbf4f2c9df3d4491b7ca54c1d065ff7f P 399a36acb0b447e8b9c0d6992bc86ea2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:44:00.217489 7905 ts_tablet_manager.cc:1905] T cbf4f2c9df3d4491b7ca54c1d065ff7f P 449f7e56e3754b22a114d8f19ed82f76: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:44:00.218411 7904 ts_tablet_manager.cc:1905] T cbf4f2c9df3d4491b7ca54c1d065ff7f P 399a36acb0b447e8b9c0d6992bc86ea2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:44:00.221482 7903 ts_tablet_manager.cc:1918] T cbf4f2c9df3d4491b7ca54c1d065ff7f P 4e87af70e6654dff8d6f88b6b739ee42: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 16:44:00.223405 7903 log.cc:1196] T cbf4f2c9df3d4491b7ca54c1d065ff7f P 4e87af70e6654dff8d6f88b6b739ee42: Deleting WAL directory at /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887385903558-388-0/minicluster-data/ts-0/wal/wals/cbf4f2c9df3d4491b7ca54c1d065ff7f
I20260619 16:44:00.223753 7905 ts_tablet_manager.cc:1918] T cbf4f2c9df3d4491b7ca54c1d065ff7f P 449f7e56e3754b22a114d8f19ed82f76: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 16:44:00.224016 7905 log.cc:1196] T cbf4f2c9df3d4491b7ca54c1d065ff7f P 449f7e56e3754b22a114d8f19ed82f76: Deleting WAL directory at /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887385903558-388-0/minicluster-data/ts-1/wal/wals/cbf4f2c9df3d4491b7ca54c1d065ff7f
I20260619 16:44:00.224041 7903 ts_tablet_manager.cc:1939] T cbf4f2c9df3d4491b7ca54c1d065ff7f P 4e87af70e6654dff8d6f88b6b739ee42: Deleting consensus metadata
I20260619 16:44:00.224483 7905 ts_tablet_manager.cc:1939] T cbf4f2c9df3d4491b7ca54c1d065ff7f P 449f7e56e3754b22a114d8f19ed82f76: Deleting consensus metadata
I20260619 16:44:00.225687 7299 catalog_manager.cc:4841] TS 449f7e56e3754b22a114d8f19ed82f76 (127.0.97.2:34331): tablet cbf4f2c9df3d4491b7ca54c1d065ff7f (table test-table [id=0688b616a16c4e5badd8cd8b009d8273]) successfully deleted
I20260619 16:44:00.225863 7904 ts_tablet_manager.cc:1918] T cbf4f2c9df3d4491b7ca54c1d065ff7f P 399a36acb0b447e8b9c0d6992bc86ea2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 16:44:00.225991 7904 log.cc:1196] T cbf4f2c9df3d4491b7ca54c1d065ff7f P 399a36acb0b447e8b9c0d6992bc86ea2: Deleting WAL directory at /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887385903558-388-0/minicluster-data/ts-2/wal/wals/cbf4f2c9df3d4491b7ca54c1d065ff7f
I20260619 16:44:00.225991 7301 catalog_manager.cc:4841] TS 4e87af70e6654dff8d6f88b6b739ee42 (127.0.97.1:40321): tablet cbf4f2c9df3d4491b7ca54c1d065ff7f (table test-table [id=0688b616a16c4e5badd8cd8b009d8273]) successfully deleted
I20260619 16:44:00.226487 7904 ts_tablet_manager.cc:1939] T cbf4f2c9df3d4491b7ca54c1d065ff7f P 399a36acb0b447e8b9c0d6992bc86ea2: Deleting consensus metadata
I20260619 16:44:00.227363 7298 catalog_manager.cc:4841] TS 399a36acb0b447e8b9c0d6992bc86ea2 (127.0.97.3:40501): tablet cbf4f2c9df3d4491b7ca54c1d065ff7f (table test-table [id=0688b616a16c4e5badd8cd8b009d8273]) successfully deleted
I20260619 16:44:00.234568 7874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:44:00.207415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33786 (local address 127.0.97.1:40321)
0619 16:44:00.207618 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0619 16:44:00.207625 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:44:00.213026 (+ 5401us) server_negotiation.cc:407] Connection header received
0619 16:44:00.213203 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:44:00.213210 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:44:00.213306 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:44:00.213417 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:44:00.215459 (+ 2042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:44:00.216464 (+ 1005us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:44:00.223349 (+ 6885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:44:00.223958 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:44:00.224086 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:44:00.228795 (+ 4709us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:44:00.228808 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:44:00.228824 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:44:00.228856 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:44:00.231595 (+ 2739us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:44:00.232409 (+ 814us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:44:00.232418 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:44:00.232420 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:44:00.232565 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:44:00.232936 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:44:00.232939 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:44:00.232941 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:44:00.233396 (+ 455us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:44:00.233678 (+ 282us) server_negotiation.cc:1035] Waiting for connection context
0619 16:44:00.234060 (+ 382us) server_negotiation.cc:299] Negotiation successful
0619 16:44:00.234217 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260619 16:44:00.235214 7488 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:44:00.348348 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu with pid 7319
I20260619 16:44:00.365850 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu pid 7508
I20260619 16:44:00.381623 7877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:44:00.366535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46564 (local address 127.0.97.2:34331)
0619 16:44:00.366709 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0619 16:44:00.366715 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:44:00.366949 (+ 234us) server_negotiation.cc:407] Connection header received
0619 16:44:00.367195 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:44:00.367200 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:44:00.367288 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:44:00.367373 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:44:00.368988 (+ 1615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:44:00.369926 (+ 938us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:44:00.371662 (+ 1736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:44:00.372084 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:44:00.372186 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:44:00.375355 (+ 3169us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:44:00.375369 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:44:00.375385 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:44:00.375433 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:44:00.378643 (+ 3210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:44:00.379500 (+ 857us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:44:00.379510 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:44:00.379514 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:44:00.379685 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:44:00.380083 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:44:00.380087 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:44:00.380089 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:44:00.380509 (+ 420us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:44:00.380753 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0619 16:44:00.381122 (+ 369us) server_negotiation.cc:299] Negotiation successful
0619 16:44:00.381308 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260619 16:44:00.382283 7631 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:44:00.453258 7858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:44:00.452751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46817 (local address 127.0.0.1:46622)
0619 16:44:00.452962 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0619 16:44:00.453078 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260619 16:44:00.454423 7858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:44:00.454023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46817 (local address 127.0.0.1:46634)
0619 16:44:00.454168 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0619 16:44:00.454261 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260619 16:44:00.474987 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu with pid 7508
I20260619 16:44:00.488819 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu pid 7657
I20260619 16:44:00.503906 7875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:44:00.489608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51258 (local address 127.0.97.3:40501)
0619 16:44:00.489801 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0619 16:44:00.489807 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:44:00.489827 (+ 20us) server_negotiation.cc:407] Connection header received
0619 16:44:00.490000 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:44:00.490006 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:44:00.490095 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:44:00.490172 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:44:00.491935 (+ 1763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:44:00.492762 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:44:00.494632 (+ 1870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:44:00.495053 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:44:00.495148 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:44:00.498271 (+ 3123us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:44:00.498283 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:44:00.498308 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:44:00.498342 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:44:00.501154 (+ 2812us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:44:00.501948 (+ 794us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:44:00.501961 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:44:00.501964 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:44:00.502121 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:44:00.502519 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:44:00.502522 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:44:00.502524 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:44:00.502883 (+ 359us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:44:00.503107 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0619 16:44:00.503421 (+ 314us) server_negotiation.cc:299] Negotiation successful
0619 16:44:00.503581 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260619 16:44:00.504670 7780 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:44:00.626384 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu with pid 7657
I20260619 16:44:00.642206 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu pid 6988
I20260619 16:44:00.643255 7154 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:44:00.754611 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu with pid 6988
I20260619 16:44:00.770216 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu pid 7087
I20260619 16:44:00.771255 7345 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:44:00.886413 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu with pid 7087
2026-06-19T16:44:00Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (8960 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887385903558-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 16:44:00 dist-test-slave-wvz7 krb5kdc[7914](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 16:44:00 dist-test-slave-wvz7 krb5kdc[7914](info): set up 2 sockets
Jun 19 16:44:00 dist-test-slave-wvz7 krb5kdc[7914](info): commencing operation
krb5kdc: starting...
W20260619 16:44:01.886412 388 mini_kdc.cc:121] Time spent starting KDC: real 0.961s user 0.002s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887385903558-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887385903558-388-0/krb5kdc/test-user.keytab.
Jun 19 16:44:02 dist-test-slave-wvz7 krb5kdc[7914](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887442, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T16:44:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T16:44:02Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887385903558-388-0/krb5kdc/oryx_127.0.97.62.keytab.
Entry for principal oryx/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887385903558-388-0/krb5kdc/oryx_127.0.97.62.keytab.
I20260619 16:44:02.076761 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu
/tmp/dist-test-taskEs2EH8/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887385903558-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887385903558-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887385903558-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887385903558-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39907
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887385903558-388-0/krb5kdc/oryx_127.0.97.62.keytab
--principal=oryx/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45439
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:39907
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887385903558-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887385903558-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887385903558-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:44:02.375450 7927 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:44:02.375779 7927 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:44:02.375907 7927 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:44:02.383050 7927 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:44:02.383160 7927 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260619 16:44:02.383203 7927 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:44:02.383289 7927 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:44:02.383349 7927 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:44:02.383432 7927 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:44:02.393312 7927 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45439
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887385903558-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887385903558-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:39907
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887385903558-388-0/krb5kdc/oryx_127.0.97.62.keytab
--principal=oryx/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39907
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887385903558-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887385903558-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:44:02.394609 7927 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:44:02.395865 7927 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:44:02.406836 7935 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:44:02.406911 7932 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:44:02.406894 7933 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:44:02.407469 7927 server_base.cc:1034] running on GCE node
I20260619 16:44:02.408246 7927 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:44:02.409488 7927 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:44:02.410733 7927 hybrid_clock.cc:648] HybridClock initialized: now 1781887442410663 us; error 37 us; skew 500 ppm
I20260619 16:44:02.411068 7927 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:44:02 dist-test-slave-wvz7 krb5kdc[7914](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887442, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:44:02.418430 7927 init.cc:377] Logged in from keytab as oryx/127.0.97.62@KRBTEST.COM (short username oryx)
I20260619 16:44:02.419795 7927 webserver.cc:460] Webserver started at http://127.0.97.62:41917/ using document root <none> and password file <none>
I20260619 16:44:02.420449 7927 fs_manager.cc:359] Metadata directory not provided
I20260619 16:44:02.420542 7927 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:44:02.420840 7927 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:44:02.423681 7927 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887385903558-388-0/minicluster-data/master-0/data/instance:
uuid: "ed5a208fb1504c2dbce0eceab08ad43c"
format_stamp: "Formatted at 2026-06-19 16:44:02 on dist-test-slave-wvz7"
server_key: "81acc01144da4eb9c1fa1b4c8e8cce08"
server_key_iv: "2299b5cee41360953542b0c479d6bd25"
server_key_version: "encryptionkey@0"
I20260619 16:44:02.424585 7927 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887385903558-388-0/minicluster-data/master-0/wal/instance:
uuid: "ed5a208fb1504c2dbce0eceab08ad43c"
format_stamp: "Formatted at 2026-06-19 16:44:02 on dist-test-slave-wvz7"
server_key: "81acc01144da4eb9c1fa1b4c8e8cce08"
server_key_iv: "2299b5cee41360953542b0c479d6bd25"
server_key_version: "encryptionkey@0"
I20260619 16:44:02.430580 7927 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260619 16:44:02.433986 7942 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:44:02.435395 7927 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260619 16:44:02.435554 7927 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887385903558-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887385903558-388-0/minicluster-data/master-0/wal
uuid: "ed5a208fb1504c2dbce0eceab08ad43c"
format_stamp: "Formatted at 2026-06-19 16:44:02 on dist-test-slave-wvz7"
server_key: "81acc01144da4eb9c1fa1b4c8e8cce08"
server_key_iv: "2299b5cee41360953542b0c479d6bd25"
server_key_version: "encryptionkey@0"
I20260619 16:44:02.435752 7927 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887385903558-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887385903558-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887385903558-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:44:02.456825 7927 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:44:02.460836 7927 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:44:02.461066 7927 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:44:02.480553 7927 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:39907
I20260619 16:44:02.480571 8003 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:39907 every 8 connection(s)
I20260619 16:44:02.481858 7927 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887385903558-388-0/minicluster-data/master-0/data/info.pb
I20260619 16:44:02.484874 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu as pid 7927
I20260619 16:44:02.485085 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887385903558-388-0/minicluster-data/master-0/wal/instance
I20260619 16:44:02.485671 8004 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:44:02.485726 388 external_mini_cluster.cc:1428] Setting key ab86ea3b6ef06493ebd03166a4a6e422
I20260619 16:44:02.499190 8004 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ed5a208fb1504c2dbce0eceab08ad43c: Bootstrap starting.
Jun 19 16:44:02 dist-test-slave-wvz7 krb5kdc[7914](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887442, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260619 16:44:02.502445 8004 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ed5a208fb1504c2dbce0eceab08ad43c: Neither blocks nor log segments found. Creating new log.
I20260619 16:44:02.503424 8004 log.cc:824] T 00000000000000000000000000000000 P ed5a208fb1504c2dbce0eceab08ad43c: Log is configured to *not* fsync() on all Append() calls
I20260619 16:44:02.506263 8004 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ed5a208fb1504c2dbce0eceab08ad43c: No bootstrap required, opened a new log
I20260619 16:44:02.507550 8007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:44:02.487733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34404 (local address 127.0.97.62:39907)
0619 16:44:02.488501 (+ 768us) server_negotiation.cc:206] Beginning negotiation
0619 16:44:02.488512 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 16:44:02.488550 (+ 38us) server_negotiation.cc:407] Connection header received
0619 16:44:02.490186 (+ 1636us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:44:02.490208 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:44:02.490597 (+ 389us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:44:02.490931 (+ 334us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:44:02.492618 (+ 1687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:44:02.493933 (+ 1315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:44:02.495593 (+ 1660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:44:02.496379 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:44:02.496536 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:44:02.500986 (+ 4450us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:44:02.500994 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:44:02.501007 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:44:02.501040 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:44:02.503860 (+ 2820us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:44:02.504494 (+ 634us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:44:02.504502 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:44:02.504508 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:44:02.504658 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:44:02.505066 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:44:02.505070 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:44:02.505072 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:44:02.505559 (+ 487us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:44:02.505806 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0619 16:44:02.506226 (+ 420us) server_negotiation.cc:299] Negotiation successful
0619 16:44:02.506616 (+ 390us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":560,"thread_start_us":343,"threads_started":1}
I20260619 16:44:02.513145 8004 raft_consensus.cc:348] T 00000000000000000000000000000000 P ed5a208fb1504c2dbce0eceab08ad43c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed5a208fb1504c2dbce0eceab08ad43c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39907 } }
I20260619 16:44:02.513448 8004 raft_consensus.cc:374] T 00000000000000000000000000000000 P ed5a208fb1504c2dbce0eceab08ad43c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:44:02.513540 8004 raft_consensus.cc:729] T 00000000000000000000000000000000 P ed5a208fb1504c2dbce0eceab08ad43c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed5a208fb1504c2dbce0eceab08ad43c, State: Initialized, Role: FOLLOWER
I20260619 16:44:02.514045 8004 consensus_queue.cc:260] T 00000000000000000000000000000000 P ed5a208fb1504c2dbce0eceab08ad43c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed5a208fb1504c2dbce0eceab08ad43c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39907 } }
I20260619 16:44:02.514252 8004 raft_consensus.cc:388] T 00000000000000000000000000000000 P ed5a208fb1504c2dbce0eceab08ad43c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:44:02.514369 8004 raft_consensus.cc:482] T 00000000000000000000000000000000 P ed5a208fb1504c2dbce0eceab08ad43c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:44:02.514518 8004 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ed5a208fb1504c2dbce0eceab08ad43c [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:44:02.516450 8004 raft_consensus.cc:504] T 00000000000000000000000000000000 P ed5a208fb1504c2dbce0eceab08ad43c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed5a208fb1504c2dbce0eceab08ad43c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39907 } }
I20260619 16:44:02.516899 8004 leader_election.cc:302] T 00000000000000000000000000000000 P ed5a208fb1504c2dbce0eceab08ad43c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ed5a208fb1504c2dbce0eceab08ad43c; no voters:
I20260619 16:44:02.517400 8004 leader_election.cc:288] T 00000000000000000000000000000000 P ed5a208fb1504c2dbce0eceab08ad43c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 16:44:02.517529 8009 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ed5a208fb1504c2dbce0eceab08ad43c [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:44:02.518548 8009 raft_consensus.cc:686] T 00000000000000000000000000000000 P ed5a208fb1504c2dbce0eceab08ad43c [term 1 LEADER]: Becoming Leader. State: Replica: ed5a208fb1504c2dbce0eceab08ad43c, State: Running, Role: LEADER
I20260619 16:44:02.519007 8009 consensus_queue.cc:237] T 00000000000000000000000000000000 P ed5a208fb1504c2dbce0eceab08ad43c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed5a208fb1504c2dbce0eceab08ad43c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39907 } }
I20260619 16:44:02.519341 8004 sys_catalog.cc:564] T 00000000000000000000000000000000 P ed5a208fb1504c2dbce0eceab08ad43c [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:44:02.521461 8011 sys_catalog.cc:455] T 00000000000000000000000000000000 P ed5a208fb1504c2dbce0eceab08ad43c [sys.catalog]: SysCatalogTable state changed. Reason: New leader ed5a208fb1504c2dbce0eceab08ad43c. Latest consensus state: current_term: 1 leader_uuid: "ed5a208fb1504c2dbce0eceab08ad43c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed5a208fb1504c2dbce0eceab08ad43c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39907 } } }
I20260619 16:44:02.521903 8011 sys_catalog.cc:458] T 00000000000000000000000000000000 P ed5a208fb1504c2dbce0eceab08ad43c [sys.catalog]: This master's current role is: LEADER
I20260619 16:44:02.521800 8010 sys_catalog.cc:455] T 00000000000000000000000000000000 P ed5a208fb1504c2dbce0eceab08ad43c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ed5a208fb1504c2dbce0eceab08ad43c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed5a208fb1504c2dbce0eceab08ad43c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39907 } } }
I20260619 16:44:02.522171 8010 sys_catalog.cc:458] T 00000000000000000000000000000000 P ed5a208fb1504c2dbce0eceab08ad43c [sys.catalog]: This master's current role is: LEADER
I20260619 16:44:02.523149 8018 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:44:02.528496 8018 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:44:02.536710 8018 catalog_manager.cc:1347] Generated new cluster ID: 1c51b93b994747f7a45b6eb0d520365b
I20260619 16:44:02.536814 8018 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:44:02.572723 8018 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 16:44:02.574116 8018 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:44:02.586671 8018 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ed5a208fb1504c2dbce0eceab08ad43c: Generated new TSK 0
I20260619 16:44:02.587541 8018 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887385903558-388-0/krb5kdc/oryx_127.0.97.1.keytab.
Entry for principal oryx/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887385903558-388-0/krb5kdc/oryx_127.0.97.1.keytab.
I20260619 16:44:02.635182 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu
/tmp/dist-test-taskEs2EH8/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887385903558-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887385903558-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887385903558-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887385903558-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39907
--keytab_file=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887385903558-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45439
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887385903558-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887385903558-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887385903558-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:44:02.937448 8030 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:44:02.937721 8030 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:44:02.937867 8030 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:44:02.945164 8030 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:44:02.945288 8030 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:44:02.945417 8030 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 16:44:02.945519 8030 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260619 16:44:02.955163 8030 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45439
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887385903558-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887385903558-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887385903558-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887385903558-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:39907
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887385903558-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:44:02.956410 8030 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:44:02.957720 8030 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:44:02.970250 8035 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:44:02.970217 8038 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:44:02.970280 8036 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:44:02.971000 8030 server_base.cc:1034] running on GCE node
I20260619 16:44:02.971585 8030 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:44:02.972489 8030 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:44:02.973716 8030 hybrid_clock.cc:648] HybridClock initialized: now 1781887442973656 us; error 73 us; skew 500 ppm
I20260619 16:44:02.974038 8030 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:44:02 dist-test-slave-wvz7 krb5kdc[7914](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887442, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:44:02.976958 8030 init.cc:377] Logged in from keytab as oryx/127.0.97.1@KRBTEST.COM (short username oryx)
I20260619 16:44:02.978333 8030 webserver.cc:460] Webserver started at http://127.0.97.1:36137/ using document root <none> and password file <none>
I20260619 16:44:02.978955 8030 fs_manager.cc:359] Metadata directory not provided
I20260619 16:44:02.979036 8030 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:44:02.979274 8030 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:44:02.982007 8030 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887385903558-388-0/minicluster-data/ts-0/data/instance:
uuid: "f9432d0f6c60491b8a20b8dcf3561c84"
format_stamp: "Formatted at 2026-06-19 16:44:02 on dist-test-slave-wvz7"
server_key: "0cc7e785ee8318a75a8026c15dec8604"
server_key_iv: "99610660aabaf674406f58fc0c14a3d3"
server_key_version: "encryptionkey@0"
I20260619 16:44:02.982689 8030 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887385903558-388-0/minicluster-data/ts-0/wal/instance:
uuid: "f9432d0f6c60491b8a20b8dcf3561c84"
format_stamp: "Formatted at 2026-06-19 16:44:02 on dist-test-slave-wvz7"
server_key: "0cc7e785ee8318a75a8026c15dec8604"
server_key_iv: "99610660aabaf674406f58fc0c14a3d3"
server_key_version: "encryptionkey@0"
I20260619 16:44:02.987488 8030 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260619 16:44:02.990826 8045 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:44:02.992206 8030 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260619 16:44:02.992345 8030 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887385903558-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887385903558-388-0/minicluster-data/ts-0/wal
uuid: "f9432d0f6c60491b8a20b8dcf3561c84"
format_stamp: "Formatted at 2026-06-19 16:44:02 on dist-test-slave-wvz7"
server_key: "0cc7e785ee8318a75a8026c15dec8604"
server_key_iv: "99610660aabaf674406f58fc0c14a3d3"
server_key_version: "encryptionkey@0"
I20260619 16:44:02.992534 8030 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887385903558-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887385903558-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887385903558-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:44:03.010293 8030 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:44:03.014173 8030 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:44:03.014415 8030 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:44:03.016928 8030 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:44:03.017009 8030 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:44:03.017123 8030 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:44:03.017220 8030 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 16:44:03 dist-test-slave-wvz7 krb5kdc[7914](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887442, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260619 16:44:03.039314 8054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:44:03.017395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:48073 (local address 127.0.97.62:39907)
0619 16:44:03.018057 (+ 662us) server_negotiation.cc:206] Beginning negotiation
0619 16:44:03.018063 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:44:03.019171 (+ 1108us) server_negotiation.cc:407] Connection header received
0619 16:44:03.021193 (+ 2022us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:44:03.021199 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:44:03.021287 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:44:03.021407 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:44:03.023603 (+ 2196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:44:03.024680 (+ 1077us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:44:03.027579 (+ 2899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:44:03.028208 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:44:03.028357 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:44:03.032782 (+ 4425us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:44:03.032788 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:44:03.032791 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:44:03.032827 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:44:03.035514 (+ 2687us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:44:03.036580 (+ 1066us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:44:03.036585 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:44:03.036587 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:44:03.036761 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:44:03.037330 (+ 569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:44:03.037334 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:44:03.037337 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:44:03.037643 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:44:03.037826 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0619 16:44:03.038815 (+ 989us) server_negotiation.cc:299] Negotiation successful
0619 16:44:03.038985 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":546,"thread_start_us":449,"threads_started":1}
I20260619 16:44:03.040894 8055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:44:03.018042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39907 (local address 127.0.97.1:48073)
0619 16:44:03.019008 (+ 966us) negotiation.cc:107] Waiting for socket to connect
0619 16:44:03.019057 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0619 16:44:03.020924 (+ 1867us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:44:03.021646 (+ 722us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:44:03.021654 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:44:03.022557 (+ 903us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:44:03.023415 (+ 858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:44:03.023444 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:44:03.024888 (+ 1444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:44:03.024898 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:44:03.027367 (+ 2469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:44:03.027385 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:44:03.028454 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:44:03.028460 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:44:03.028638 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:44:03.029369 (+ 731us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:44:03.029393 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:44:03.032441 (+ 3048us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:44:03.035666 (+ 3225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:44:03.035678 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:44:03.035700 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:44:03.036372 (+ 672us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:44:03.036902 (+ 530us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:44:03.036908 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:44:03.036912 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:44:03.037190 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:44:03.037791 (+ 601us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:44:03.037799 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:44:03.038331 (+ 532us) client_negotiation.cc:769] Sending connection context
0619 16:44:03.038684 (+ 353us) client_negotiation.cc:240] Negotiation successful
0619 16:44:03.039320 (+ 636us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":672,"thread_start_us":349,"threads_started":1}
I20260619 16:44:03.058531 8109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:44:03.046640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39907 (local address 127.0.97.1:44953)
0619 16:44:03.047217 (+ 577us) negotiation.cc:107] Waiting for socket to connect
0619 16:44:03.047232 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 16:44:03.047355 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:44:03.047641 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:44:03.047645 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:44:03.048054 (+ 409us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:44:03.048324 (+ 270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:44:03.048333 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:44:03.049247 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:44:03.049252 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:44:03.050457 (+ 1205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:44:03.050468 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:44:03.051200 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:44:03.051206 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:44:03.051364 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:44:03.051963 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:44:03.051978 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:44:03.052642 (+ 664us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:44:03.055846 (+ 3204us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:44:03.055850 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:44:03.055852 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:44:03.056277 (+ 425us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:44:03.056688 (+ 411us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:44:03.056693 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:44:03.056696 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:44:03.056827 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:44:03.057364 (+ 537us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:44:03.057368 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:44:03.057583 (+ 215us) client_negotiation.cc:769] Sending connection context
0619 16:44:03.057778 (+ 195us) client_negotiation.cc:240] Negotiation successful
0619 16:44:03.058054 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":445,"thread_start_us":318,"threads_started":1}
I20260619 16:44:03.059199 8054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:44:03.046665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:44953 (local address 127.0.97.62:39907)
0619 16:44:03.046813 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0619 16:44:03.046816 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 16:44:03.047313 (+ 497us) server_negotiation.cc:407] Connection header received
0619 16:44:03.047458 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:44:03.047463 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:44:03.047532 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:44:03.047608 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:44:03.048449 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:44:03.049119 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:44:03.050573 (+ 1454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:44:03.051027 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:44:03.051131 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:44:03.052795 (+ 1664us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:44:03.052800 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:44:03.052804 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:44:03.052831 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:44:03.055728 (+ 2897us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:44:03.056391 (+ 663us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:44:03.056405 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:44:03.056409 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:44:03.056559 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:44:03.056927 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:44:03.056932 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:44:03.056936 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:44:03.057265 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:44:03.057400 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0619 16:44:03.058835 (+ 1435us) server_negotiation.cc:299] Negotiation successful
0619 16:44:03.058988 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260619 16:44:03.070963 8030 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:39607
I20260619 16:44:03.070967 8165 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:39607 every 8 connection(s)
I20260619 16:44:03.072085 8030 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887385903558-388-0/minicluster-data/ts-0/data/info.pb
I20260619 16:44:03.073258 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu as pid 8030
I20260619 16:44:03.073442 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887385903558-388-0/minicluster-data/ts-0/wal/instance
I20260619 16:44:03.073866 8166 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39907
I20260619 16:44:03.074064 8166 heartbeater.cc:464] Registering TS with master...
I20260619 16:44:03.074162 388 external_mini_cluster.cc:1428] Setting key 26edcdafc4a9328d70aa0ceb77c6ac2e
I20260619 16:44:03.074680 8166 heartbeater.cc:511] Master 127.0.97.62:39907 requested a full tablet report, sending...
I20260619 16:44:03.076403 7959 ts_manager.cc:194] Registered new tserver with Master: f9432d0f6c60491b8a20b8dcf3561c84 (127.0.97.1:39607)
I20260619 16:44:03.077759 7959 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:48073
WARNING: no policy specified for oryx/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887385903558-388-0/krb5kdc/oryx_127.0.97.2.keytab.
Entry for principal oryx/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887385903558-388-0/krb5kdc/oryx_127.0.97.2.keytab.
I20260619 16:44:03.111866 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu
/tmp/dist-test-taskEs2EH8/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887385903558-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887385903558-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887385903558-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887385903558-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39907
--keytab_file=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887385903558-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45439
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887385903558-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887385903558-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887385903558-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:44:03.404982 8170 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:44:03.405230 8170 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:44:03.405328 8170 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:44:03.412530 8170 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:44:03.412634 8170 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:44:03.412729 8170 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 16:44:03.412806 8170 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260619 16:44:03.422226 8170 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45439
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887385903558-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887385903558-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887385903558-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887385903558-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:39907
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887385903558-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:44:03.423358 8170 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:44:03.424564 8170 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:44:03.437183 8176 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:44:03.437186 8175 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:44:03.437245 8178 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:44:03.438167 8170 server_base.cc:1034] running on GCE node
I20260619 16:44:03.438683 8170 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:44:03.439513 8170 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:44:03.440716 8170 hybrid_clock.cc:648] HybridClock initialized: now 1781887443440641 us; error 41 us; skew 500 ppm
I20260619 16:44:03.440970 8170 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:44:03 dist-test-slave-wvz7 krb5kdc[7914](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887443, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:44:03.443825 8170 init.cc:377] Logged in from keytab as oryx/127.0.97.2@KRBTEST.COM (short username oryx)
I20260619 16:44:03.445025 8170 webserver.cc:460] Webserver started at http://127.0.97.2:43089/ using document root <none> and password file <none>
I20260619 16:44:03.445622 8170 fs_manager.cc:359] Metadata directory not provided
I20260619 16:44:03.445708 8170 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:44:03.446018 8170 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:44:03.448625 8170 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887385903558-388-0/minicluster-data/ts-1/data/instance:
uuid: "410592fd61c042c9a956aada0e560a12"
format_stamp: "Formatted at 2026-06-19 16:44:03 on dist-test-slave-wvz7"
server_key: "9afb93178b630f0878db5a876f8719da"
server_key_iv: "15874e83c90cae7089f128937aacfbb1"
server_key_version: "encryptionkey@0"
I20260619 16:44:03.449323 8170 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887385903558-388-0/minicluster-data/ts-1/wal/instance:
uuid: "410592fd61c042c9a956aada0e560a12"
format_stamp: "Formatted at 2026-06-19 16:44:03 on dist-test-slave-wvz7"
server_key: "9afb93178b630f0878db5a876f8719da"
server_key_iv: "15874e83c90cae7089f128937aacfbb1"
server_key_version: "encryptionkey@0"
I20260619 16:44:03.453837 8170 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.003s
I20260619 16:44:03.457247 8185 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:44:03.458710 8170 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260619 16:44:03.458868 8170 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887385903558-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887385903558-388-0/minicluster-data/ts-1/wal
uuid: "410592fd61c042c9a956aada0e560a12"
format_stamp: "Formatted at 2026-06-19 16:44:03 on dist-test-slave-wvz7"
server_key: "9afb93178b630f0878db5a876f8719da"
server_key_iv: "15874e83c90cae7089f128937aacfbb1"
server_key_version: "encryptionkey@0"
I20260619 16:44:03.459079 8170 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887385903558-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887385903558-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887385903558-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:44:03.482812 8170 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:44:03.487815 8170 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:44:03.488066 8170 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:44:03.491139 8170 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:44:03.491236 8170 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:44:03.491402 8170 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:44:03.491483 8170 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 16:44:03 dist-test-slave-wvz7 krb5kdc[7914](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887443, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260619 16:44:03.511845 8054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:44:03.491547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:38633 (local address 127.0.97.62:39907)
0619 16:44:03.491728 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0619 16:44:03.491734 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:44:03.493123 (+ 1389us) server_negotiation.cc:407] Connection header received
0619 16:44:03.495201 (+ 2078us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:44:03.495206 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:44:03.495286 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:44:03.495385 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:44:03.497557 (+ 2172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:44:03.498691 (+ 1134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:44:03.501353 (+ 2662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:44:03.501913 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:44:03.502020 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:44:03.505355 (+ 3335us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:44:03.505360 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:44:03.505363 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:44:03.505392 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:44:03.508644 (+ 3252us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:44:03.509541 (+ 897us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:44:03.509547 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:44:03.509551 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:44:03.509705 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:44:03.510150 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:44:03.510155 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:44:03.510159 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:44:03.510410 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:44:03.510546 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0619 16:44:03.511311 (+ 765us) server_negotiation.cc:299] Negotiation successful
0619 16:44:03.511519 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260619 16:44:03.513226 8194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:44:03.492203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39907 (local address 127.0.97.2:38633)
0619 16:44:03.492972 (+ 769us) negotiation.cc:107] Waiting for socket to connect
0619 16:44:03.493020 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0619 16:44:03.494903 (+ 1883us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:44:03.495852 (+ 949us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:44:03.495869 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:44:03.496624 (+ 755us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:44:03.497405 (+ 781us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:44:03.497422 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:44:03.498889 (+ 1467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:44:03.498895 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:44:03.501155 (+ 2260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:44:03.501173 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:44:03.502146 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:44:03.502154 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:44:03.502340 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:44:03.502919 (+ 579us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:44:03.502947 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:44:03.505144 (+ 2197us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:44:03.508767 (+ 3623us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:44:03.508776 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:44:03.508790 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:44:03.509400 (+ 610us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:44:03.509866 (+ 466us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:44:03.509869 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:44:03.509871 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:44:03.510049 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:44:03.510548 (+ 499us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:44:03.510556 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:44:03.510926 (+ 370us) client_negotiation.cc:769] Sending connection context
0619 16:44:03.511241 (+ 315us) client_negotiation.cc:240] Negotiation successful
0619 16:44:03.511727 (+ 486us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":514,"thread_start_us":296,"threads_started":1}
I20260619 16:44:03.531307 8280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:44:03.519588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39907 (local address 127.0.97.2:59725)
0619 16:44:03.520123 (+ 535us) negotiation.cc:107] Waiting for socket to connect
0619 16:44:03.520141 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 16:44:03.520291 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:44:03.520578 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:44:03.520583 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:44:03.521156 (+ 573us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:44:03.521532 (+ 376us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:44:03.521548 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:44:03.522608 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:44:03.522614 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:44:03.523688 (+ 1074us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:44:03.523702 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:44:03.524681 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:44:03.524685 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:44:03.524812 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:44:03.525479 (+ 667us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:44:03.525496 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:44:03.526409 (+ 913us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:44:03.529027 (+ 2618us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:44:03.529033 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:44:03.529037 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:44:03.529385 (+ 348us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:44:03.529698 (+ 313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:44:03.529703 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:44:03.529707 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:44:03.529868 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:44:03.530360 (+ 492us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:44:03.530364 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:44:03.530541 (+ 177us) client_negotiation.cc:769] Sending connection context
0619 16:44:03.530701 (+ 160us) client_negotiation.cc:240] Negotiation successful
0619 16:44:03.530993 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":427,"thread_start_us":304,"threads_started":1}
I20260619 16:44:03.531919 8054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:44:03.519572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:59725 (local address 127.0.97.62:39907)
0619 16:44:03.519737 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0619 16:44:03.519741 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:44:03.520281 (+ 540us) server_negotiation.cc:407] Connection header received
0619 16:44:03.520397 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:44:03.520402 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:44:03.520463 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:44:03.520538 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:44:03.521724 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:44:03.522486 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:44:03.523828 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:44:03.524511 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:44:03.524647 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:44:03.526602 (+ 1955us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:44:03.526607 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:44:03.526610 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:44:03.526639 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:44:03.528914 (+ 2275us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:44:03.529485 (+ 571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:44:03.529489 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:44:03.529492 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:44:03.529602 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:44:03.529960 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:44:03.529965 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:44:03.529968 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:44:03.530243 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:44:03.530367 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0619 16:44:03.530995 (+ 628us) server_negotiation.cc:299] Negotiation successful
0619 16:44:03.531266 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260619 16:44:03.537384 8170 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:45795
I20260619 16:44:03.537400 8304 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:45795 every 8 connection(s)
I20260619 16:44:03.538646 8170 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887385903558-388-0/minicluster-data/ts-1/data/info.pb
I20260619 16:44:03.540464 8305 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39907
I20260619 16:44:03.540664 8305 heartbeater.cc:464] Registering TS with master...
I20260619 16:44:03.541235 8305 heartbeater.cc:511] Master 127.0.97.62:39907 requested a full tablet report, sending...
I20260619 16:44:03.542583 7959 ts_manager.cc:194] Registered new tserver with Master: 410592fd61c042c9a956aada0e560a12 (127.0.97.2:45795)
I20260619 16:44:03.543417 7959 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:38633
I20260619 16:44:03.547962 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu as pid 8170
I20260619 16:44:03.548115 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887385903558-388-0/minicluster-data/ts-1/wal/instance
I20260619 16:44:03.548547 388 external_mini_cluster.cc:1428] Setting key b0d1b93da149252252f170ad45ad33f0
WARNING: no policy specified for oryx/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887385903558-388-0/krb5kdc/oryx_127.0.97.3.keytab.
Entry for principal oryx/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887385903558-388-0/krb5kdc/oryx_127.0.97.3.keytab.
I20260619 16:44:03.586292 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu
/tmp/dist-test-taskEs2EH8/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887385903558-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887385903558-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887385903558-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887385903558-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39907
--keytab_file=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887385903558-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45439
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887385903558-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887385903558-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887385903558-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:44:03.881069 8309 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:44:03.881327 8309 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:44:03.881474 8309 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:44:03.888690 8309 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:44:03.888818 8309 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:44:03.888942 8309 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 16:44:03.889061 8309 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260619 16:44:03.899806 8309 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45439
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887385903558-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887385903558-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887385903558-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887385903558-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:39907
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887385903558-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:44:03.900914 8309 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:44:03.902138 8309 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:44:03.913641 8314 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:44:03.913669 8317 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:44:03.913698 8315 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:44:03.914335 8309 server_base.cc:1034] running on GCE node
I20260619 16:44:03.914839 8309 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:44:03.915675 8309 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:44:03.916891 8309 hybrid_clock.cc:648] HybridClock initialized: now 1781887443916814 us; error 45 us; skew 500 ppm
I20260619 16:44:03.917150 8309 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:44:03 dist-test-slave-wvz7 krb5kdc[7914](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887443, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:44:03.919922 8309 init.cc:377] Logged in from keytab as oryx/127.0.97.3@KRBTEST.COM (short username oryx)
I20260619 16:44:03.921147 8309 webserver.cc:460] Webserver started at http://127.0.97.3:34369/ using document root <none> and password file <none>
I20260619 16:44:03.921716 8309 fs_manager.cc:359] Metadata directory not provided
I20260619 16:44:03.921841 8309 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:44:03.922119 8309 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:44:03.924697 8309 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887385903558-388-0/minicluster-data/ts-2/data/instance:
uuid: "2ebc7147b38d42c183f6c7b6a62db075"
format_stamp: "Formatted at 2026-06-19 16:44:03 on dist-test-slave-wvz7"
server_key: "e11f812d3d0f3aa9482432d9da52cbc6"
server_key_iv: "c427358ab516d35a5116244978113416"
server_key_version: "encryptionkey@0"
I20260619 16:44:03.925402 8309 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887385903558-388-0/minicluster-data/ts-2/wal/instance:
uuid: "2ebc7147b38d42c183f6c7b6a62db075"
format_stamp: "Formatted at 2026-06-19 16:44:03 on dist-test-slave-wvz7"
server_key: "e11f812d3d0f3aa9482432d9da52cbc6"
server_key_iv: "c427358ab516d35a5116244978113416"
server_key_version: "encryptionkey@0"
I20260619 16:44:03.929724 8309 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.001s
I20260619 16:44:03.932838 8324 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:44:03.934134 8309 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 16:44:03.934258 8309 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887385903558-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887385903558-388-0/minicluster-data/ts-2/wal
uuid: "2ebc7147b38d42c183f6c7b6a62db075"
format_stamp: "Formatted at 2026-06-19 16:44:03 on dist-test-slave-wvz7"
server_key: "e11f812d3d0f3aa9482432d9da52cbc6"
server_key_iv: "c427358ab516d35a5116244978113416"
server_key_version: "encryptionkey@0"
I20260619 16:44:03.934444 8309 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887385903558-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887385903558-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887385903558-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:44:03.947593 8309 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:44:03.951349 8309 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:44:03.951579 8309 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:44:03.953980 8309 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:44:03.954057 8309 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:44:03.954171 8309 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:44:03.954226 8309 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 16:44:03 dist-test-slave-wvz7 krb5kdc[7914](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887443, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260619 16:44:03.975950 8054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:44:03.954551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:48037 (local address 127.0.97.62:39907)
0619 16:44:03.954720 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0619 16:44:03.954726 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:44:03.956033 (+ 1307us) server_negotiation.cc:407] Connection header received
0619 16:44:03.958369 (+ 2336us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:44:03.958375 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:44:03.958462 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:44:03.958532 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:44:03.960793 (+ 2261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:44:03.961769 (+ 976us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:44:03.964114 (+ 2345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:44:03.964753 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:44:03.964890 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:44:03.968811 (+ 3921us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:44:03.968819 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:44:03.968825 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:44:03.968870 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:44:03.972152 (+ 3282us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:44:03.973051 (+ 899us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:44:03.973056 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:44:03.973060 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:44:03.973214 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:44:03.973675 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:44:03.973681 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:44:03.973685 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:44:03.974016 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:44:03.974169 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0619 16:44:03.975335 (+ 1166us) server_negotiation.cc:299] Negotiation successful
0619 16:44:03.975568 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260619 16:44:03.977680 8333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:44:03.955070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39907 (local address 127.0.97.3:48037)
0619 16:44:03.955869 (+ 799us) negotiation.cc:107] Waiting for socket to connect
0619 16:44:03.955913 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0619 16:44:03.958054 (+ 2141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:44:03.959023 (+ 969us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:44:03.959037 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:44:03.959763 (+ 726us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:44:03.960618 (+ 855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:44:03.960641 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:44:03.961954 (+ 1313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:44:03.961959 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:44:03.963946 (+ 1987us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:44:03.963979 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:44:03.964996 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:44:03.965004 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:44:03.965309 (+ 305us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:44:03.966080 (+ 771us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:44:03.966109 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:44:03.968569 (+ 2460us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:44:03.972303 (+ 3734us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:44:03.972312 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:44:03.972329 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:44:03.972925 (+ 596us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:44:03.973338 (+ 413us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:44:03.973344 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:44:03.973347 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:44:03.973559 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:44:03.974169 (+ 610us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:44:03.974186 (+ 17us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:44:03.974774 (+ 588us) client_negotiation.cc:769] Sending connection context
0619 16:44:03.975218 (+ 444us) client_negotiation.cc:240] Negotiation successful
0619 16:44:03.975874 (+ 656us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":524,"thread_start_us":320,"threads_started":1}
I20260619 16:44:03.992688 8309 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:46789
I20260619 16:44:03.992707 8443 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:46789 every 8 connection(s)
I20260619 16:44:03.993968 8309 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887385903558-388-0/minicluster-data/ts-2/data/info.pb
I20260619 16:44:03.995927 8444 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39907
I20260619 16:44:03.996134 8444 heartbeater.cc:464] Registering TS with master...
I20260619 16:44:03.996779 8444 heartbeater.cc:511] Master 127.0.97.62:39907 requested a full tablet report, sending...
I20260619 16:44:03.998042 7959 ts_manager.cc:194] Registered new tserver with Master: 2ebc7147b38d42c183f6c7b6a62db075 (127.0.97.3:46789)
I20260619 16:44:03.998162 8420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:44:03.984510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39907 (local address 127.0.97.3:58191)
0619 16:44:03.985159 (+ 649us) negotiation.cc:107] Waiting for socket to connect
0619 16:44:03.985175 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 16:44:03.985305 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:44:03.985618 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:44:03.985623 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:44:03.986215 (+ 592us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:44:03.986799 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:44:03.986815 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:44:03.988073 (+ 1258us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:44:03.988078 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:44:03.989716 (+ 1638us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:44:03.989775 (+ 59us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:44:03.990839 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:44:03.990845 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:44:03.990964 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:44:03.991573 (+ 609us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:44:03.991598 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:44:03.992549 (+ 951us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:44:03.995597 (+ 3048us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:44:03.995602 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:44:03.995606 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:44:03.996116 (+ 510us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:44:03.996462 (+ 346us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:44:03.996467 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:44:03.996471 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:44:03.996593 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:44:03.997106 (+ 513us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:44:03.997110 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:44:03.997279 (+ 169us) client_negotiation.cc:769] Sending connection context
0619 16:44:03.997398 (+ 119us) client_negotiation.cc:240] Negotiation successful
0619 16:44:03.997701 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":528,"thread_start_us":399,"threads_started":1}
I20260619 16:44:03.998207 8054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:44:03.984458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:58191 (local address 127.0.97.62:39907)
0619 16:44:03.984612 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0619 16:44:03.984618 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:44:03.985258 (+ 640us) server_negotiation.cc:407] Connection header received
0619 16:44:03.985432 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:44:03.985438 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:44:03.985503 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:44:03.985583 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:44:03.986943 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:44:03.987912 (+ 969us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:44:03.989899 (+ 1987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:44:03.990461 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:44:03.990579 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:44:03.992730 (+ 2151us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:44:03.992736 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:44:03.992739 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:44:03.992781 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:44:03.995484 (+ 2703us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:44:03.996228 (+ 744us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:44:03.996233 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:44:03.996235 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:44:03.996363 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:44:03.996689 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:44:03.996694 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:44:03.996698 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:44:03.997002 (+ 304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:44:03.997126 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0619 16:44:03.997511 (+ 385us) server_negotiation.cc:299] Negotiation successful
0619 16:44:03.997785 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260619 16:44:03.999078 7959 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:48037
I20260619 16:44:04.002648 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu as pid 8309
I20260619 16:44:04.002789 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887385903558-388-0/minicluster-data/ts-2/wal/instance
I20260619 16:44:04.003180 388 external_mini_cluster.cc:1428] Setting key cb35ab0717251083620e18f3f078e1ec
I20260619 16:44:04.005623 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 19 16:44:04 dist-test-slave-wvz7 krb5kdc[7914](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887444, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 19 16:44:04 dist-test-slave-wvz7 krb5kdc[7914](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:44:04 dist-test-slave-wvz7 krb5kdc[7914](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
W20260619 16:44:04.049881 8054 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:44:04.040025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48016 (local address 127.0.97.62:39907)
0619 16:44:04.040237 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0619 16:44:04.040244 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:44:04.040550 (+ 306us) server_negotiation.cc:407] Connection header received
0619 16:44:04.040693 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:44:04.040697 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:44:04.040777 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:44:04.040861 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:44:04.042125 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:44:04.042984 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:44:04.044283 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:44:04.044723 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:44:04.044854 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:44:04.049614 (+ 4760us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48016: BlockingRecv error: recv got EOF from 127.0.0.1:48016 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Jun 19 16:44:04 dist-test-slave-wvz7 krb5kdc[7914](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887444, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260619 16:44:04.065258 8054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:44:04.052870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48018 (local address 127.0.97.62:39907)
0619 16:44:04.053042 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0619 16:44:04.053048 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:44:04.053289 (+ 241us) server_negotiation.cc:407] Connection header received
0619 16:44:04.053417 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:44:04.053420 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:44:04.053489 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:44:04.053560 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:44:04.054700 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:44:04.055381 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:44:04.056535 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:44:04.056912 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:44:04.057012 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:44:04.060431 (+ 3419us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:44:04.060437 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:44:04.060440 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:44:04.060471 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:44:04.062999 (+ 2528us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:44:04.063710 (+ 711us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:44:04.063714 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:44:04.063716 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:44:04.063848 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:44:04.064173 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:44:04.064177 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:44:04.064179 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:44:04.064442 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:44:04.064647 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0619 16:44:04.064859 (+ 212us) server_negotiation.cc:299] Negotiation successful
0619 16:44:04.065026 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260619 16:44:04.069113 7959 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48018:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 16:44:04.075665 7959 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 16:44:04.080189 8166 heartbeater.cc:503] Master 127.0.97.62:39907 was elected leader, sending a full tablet report...
I20260619 16:44:04.099888 8463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:44:04.087214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40886 (local address 127.0.97.1:39607)
0619 16:44:04.087691 (+ 477us) server_negotiation.cc:206] Beginning negotiation
0619 16:44:04.087702 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 16:44:04.089027 (+ 1325us) server_negotiation.cc:407] Connection header received
0619 16:44:04.089367 (+ 340us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:44:04.089377 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:44:04.089667 (+ 290us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:44:04.089884 (+ 217us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:44:04.090984 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:44:04.092836 (+ 1852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:44:04.096549 (+ 3713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:44:04.098169 (+ 1620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:44:04.098289 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:44:04.098513 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0619 16:44:04.099365 (+ 852us) server_negotiation.cc:299] Negotiation successful
0619 16:44:04.099557 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":322,"thread_start_us":199,"threads_started":1}
I20260619 16:44:04.100988 8464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:44:04.086857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39607 (local address 127.0.0.1:40886)
0619 16:44:04.088873 (+ 2016us) negotiation.cc:107] Waiting for socket to connect
0619 16:44:04.088908 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0619 16:44:04.089101 (+ 193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:44:04.090238 (+ 1137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:44:04.090248 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:44:04.090284 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:44:04.090749 (+ 465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:44:04.090764 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:44:04.093094 (+ 2330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:44:04.093101 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:44:04.096353 (+ 3252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:44:04.096371 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:44:04.098762 (+ 2391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:44:04.098769 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:44:04.099010 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:44:04.099133 (+ 123us) client_negotiation.cc:769] Sending connection context
0619 16:44:04.100338 (+ 1205us) client_negotiation.cc:240] Negotiation successful
0619 16:44:04.100602 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1759,"thread_start_us":12022,"threads_started":1}
I20260619 16:44:04.105283 8096 tablet_service.cc:1467] Processing CreateTablet for tablet d1ea75f272c54ab3a32c2faeb373ec97 (DEFAULT_TABLE table=test-table [id=0daaa89c199e4b7abd1cf1dfb7caa624]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:44:04.106736 8096 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d1ea75f272c54ab3a32c2faeb373ec97. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:44:04.108443 8462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:44:04.087217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:46789 (local address 127.0.0.1:60330)
0619 16:44:04.098648 (+ 11431us) negotiation.cc:107] Waiting for socket to connect
0619 16:44:04.098673 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0619 16:44:04.098806 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:44:04.100233 (+ 1427us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:44:04.100238 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:44:04.100262 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:44:04.100717 (+ 455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:44:04.100730 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:44:04.103221 (+ 2491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:44:04.103226 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:44:04.105028 (+ 1802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:44:04.105041 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:44:04.107381 (+ 2340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:44:04.107387 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:44:04.107842 (+ 455us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:44:04.107873 (+ 31us) client_negotiation.cc:769] Sending connection context
0619 16:44:04.107951 (+ 78us) client_negotiation.cc:240] Negotiation successful
0619 16:44:04.108069 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11285,"mutex_wait_us":330,"thread_start_us":1265,"threads_started":1}
I20260619 16:44:04.108536 8465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:44:04.087717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60330 (local address 127.0.97.3:46789)
0619 16:44:04.088485 (+ 768us) server_negotiation.cc:206] Beginning negotiation
0619 16:44:04.088494 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 16:44:04.098786 (+ 10292us) server_negotiation.cc:407] Connection header received
0619 16:44:04.098941 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:44:04.098948 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:44:04.099205 (+ 257us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:44:04.099407 (+ 202us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:44:04.100951 (+ 1544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:44:04.103046 (+ 2095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:44:04.105173 (+ 2127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:44:04.106556 (+ 1383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:44:04.107029 (+ 473us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:44:04.107242 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0619 16:44:04.108027 (+ 785us) server_negotiation.cc:299] Negotiation successful
0619 16:44:04.108183 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":588,"thread_start_us":418,"threads_started":1}
I20260619 16:44:04.109982 8466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:44:04.089295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:45795 (local address 127.0.0.1:59360)
0619 16:44:04.103442 (+ 14147us) negotiation.cc:107] Waiting for socket to connect
0619 16:44:04.103463 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 16:44:04.103651 (+ 188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:44:04.104386 (+ 735us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:44:04.104390 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:44:04.104412 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:44:04.104764 (+ 352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:44:04.104785 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:44:04.106064 (+ 1279us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:44:04.106067 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:44:04.107669 (+ 1602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:44:04.107679 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:44:04.109235 (+ 1556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:44:04.109241 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:44:04.109394 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:44:04.109427 (+ 33us) client_negotiation.cc:769] Sending connection context
0619 16:44:04.109507 (+ 80us) client_negotiation.cc:240] Negotiation successful
0619 16:44:04.109623 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14025,"mutex_wait_us":1,"thread_start_us":13893,"threads_started":1}
I20260619 16:44:04.111967 8374 tablet_service.cc:1467] Processing CreateTablet for tablet d1ea75f272c54ab3a32c2faeb373ec97 (DEFAULT_TABLE table=test-table [id=0daaa89c199e4b7abd1cf1dfb7caa624]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:44:04.113072 8374 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d1ea75f272c54ab3a32c2faeb373ec97. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:44:04.114828 8467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:44:04.089516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59360 (local address 127.0.97.2:45795)
0619 16:44:04.102895 (+ 13379us) server_negotiation.cc:206] Beginning negotiation
0619 16:44:04.102903 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 16:44:04.103554 (+ 651us) server_negotiation.cc:407] Connection header received
0619 16:44:04.103834 (+ 280us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:44:04.103843 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:44:04.104075 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:44:04.104228 (+ 153us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:44:04.104943 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:44:04.105919 (+ 976us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:44:04.107806 (+ 1887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:44:04.109068 (+ 1262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:44:04.109173 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:44:04.109367 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0619 16:44:04.109564 (+ 197us) server_negotiation.cc:299] Negotiation successful
0619 16:44:04.114462 (+ 4898us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13229,"thread_start_us":13380,"threads_started":1}
I20260619 16:44:04.118207 8235 tablet_service.cc:1467] Processing CreateTablet for tablet d1ea75f272c54ab3a32c2faeb373ec97 (DEFAULT_TABLE table=test-table [id=0daaa89c199e4b7abd1cf1dfb7caa624]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:44:04.119550 8235 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d1ea75f272c54ab3a32c2faeb373ec97. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:44:04.119808 8468 tablet_bootstrap.cc:492] T d1ea75f272c54ab3a32c2faeb373ec97 P f9432d0f6c60491b8a20b8dcf3561c84: Bootstrap starting.
I20260619 16:44:04.123147 8468 tablet_bootstrap.cc:654] T d1ea75f272c54ab3a32c2faeb373ec97 P f9432d0f6c60491b8a20b8dcf3561c84: Neither blocks nor log segments found. Creating new log.
I20260619 16:44:04.124301 8468 log.cc:824] T d1ea75f272c54ab3a32c2faeb373ec97 P f9432d0f6c60491b8a20b8dcf3561c84: Log is configured to *not* fsync() on all Append() calls
I20260619 16:44:04.126926 8468 tablet_bootstrap.cc:492] T d1ea75f272c54ab3a32c2faeb373ec97 P f9432d0f6c60491b8a20b8dcf3561c84: No bootstrap required, opened a new log
I20260619 16:44:04.127075 8470 tablet_bootstrap.cc:492] T d1ea75f272c54ab3a32c2faeb373ec97 P 2ebc7147b38d42c183f6c7b6a62db075: Bootstrap starting.
I20260619 16:44:04.127152 8468 ts_tablet_manager.cc:1397] T d1ea75f272c54ab3a32c2faeb373ec97 P f9432d0f6c60491b8a20b8dcf3561c84: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260619 16:44:04.130328 8470 tablet_bootstrap.cc:654] T d1ea75f272c54ab3a32c2faeb373ec97 P 2ebc7147b38d42c183f6c7b6a62db075: Neither blocks nor log segments found. Creating new log.
I20260619 16:44:04.131471 8470 log.cc:824] T d1ea75f272c54ab3a32c2faeb373ec97 P 2ebc7147b38d42c183f6c7b6a62db075: Log is configured to *not* fsync() on all Append() calls
I20260619 16:44:04.132261 8471 tablet_bootstrap.cc:492] T d1ea75f272c54ab3a32c2faeb373ec97 P 410592fd61c042c9a956aada0e560a12: Bootstrap starting.
I20260619 16:44:04.135604 8468 raft_consensus.cc:348] T d1ea75f272c54ab3a32c2faeb373ec97 P f9432d0f6c60491b8a20b8dcf3561c84 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9432d0f6c60491b8a20b8dcf3561c84" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39607 } } peers { permanent_uuid: "2ebc7147b38d42c183f6c7b6a62db075" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46789 } } peers { permanent_uuid: "410592fd61c042c9a956aada0e560a12" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45795 } }
I20260619 16:44:04.136089 8471 tablet_bootstrap.cc:654] T d1ea75f272c54ab3a32c2faeb373ec97 P 410592fd61c042c9a956aada0e560a12: Neither blocks nor log segments found. Creating new log.
I20260619 16:44:04.136112 8470 tablet_bootstrap.cc:492] T d1ea75f272c54ab3a32c2faeb373ec97 P 2ebc7147b38d42c183f6c7b6a62db075: No bootstrap required, opened a new log
I20260619 16:44:04.136435 8470 ts_tablet_manager.cc:1397] T d1ea75f272c54ab3a32c2faeb373ec97 P 2ebc7147b38d42c183f6c7b6a62db075: Time spent bootstrapping tablet: real 0.011s user 0.008s sys 0.000s
I20260619 16:44:04.137234 8471 log.cc:824] T d1ea75f272c54ab3a32c2faeb373ec97 P 410592fd61c042c9a956aada0e560a12: Log is configured to *not* fsync() on all Append() calls
I20260619 16:44:04.138265 8468 raft_consensus.cc:374] T d1ea75f272c54ab3a32c2faeb373ec97 P f9432d0f6c60491b8a20b8dcf3561c84 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:44:04.138403 8468 raft_consensus.cc:729] T d1ea75f272c54ab3a32c2faeb373ec97 P f9432d0f6c60491b8a20b8dcf3561c84 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f9432d0f6c60491b8a20b8dcf3561c84, State: Initialized, Role: FOLLOWER
I20260619 16:44:04.139207 8468 consensus_queue.cc:260] T d1ea75f272c54ab3a32c2faeb373ec97 P f9432d0f6c60491b8a20b8dcf3561c84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9432d0f6c60491b8a20b8dcf3561c84" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39607 } } peers { permanent_uuid: "2ebc7147b38d42c183f6c7b6a62db075" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46789 } } peers { permanent_uuid: "410592fd61c042c9a956aada0e560a12" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45795 } }
I20260619 16:44:04.143585 8468 ts_tablet_manager.cc:1428] T d1ea75f272c54ab3a32c2faeb373ec97 P f9432d0f6c60491b8a20b8dcf3561c84: Time spent starting tablet: real 0.016s user 0.010s sys 0.000s
I20260619 16:44:04.147238 8471 tablet_bootstrap.cc:492] T d1ea75f272c54ab3a32c2faeb373ec97 P 410592fd61c042c9a956aada0e560a12: No bootstrap required, opened a new log
I20260619 16:44:04.147089 8470 raft_consensus.cc:348] T d1ea75f272c54ab3a32c2faeb373ec97 P 2ebc7147b38d42c183f6c7b6a62db075 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9432d0f6c60491b8a20b8dcf3561c84" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39607 } } peers { permanent_uuid: "2ebc7147b38d42c183f6c7b6a62db075" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46789 } } peers { permanent_uuid: "410592fd61c042c9a956aada0e560a12" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45795 } }
I20260619 16:44:04.147462 8470 raft_consensus.cc:374] T d1ea75f272c54ab3a32c2faeb373ec97 P 2ebc7147b38d42c183f6c7b6a62db075 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:44:04.147514 8471 ts_tablet_manager.cc:1397] T d1ea75f272c54ab3a32c2faeb373ec97 P 410592fd61c042c9a956aada0e560a12: Time spent bootstrapping tablet: real 0.015s user 0.005s sys 0.004s
I20260619 16:44:04.147559 8470 raft_consensus.cc:729] T d1ea75f272c54ab3a32c2faeb373ec97 P 2ebc7147b38d42c183f6c7b6a62db075 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ebc7147b38d42c183f6c7b6a62db075, State: Initialized, Role: FOLLOWER
I20260619 16:44:04.148155 8470 consensus_queue.cc:260] T d1ea75f272c54ab3a32c2faeb373ec97 P 2ebc7147b38d42c183f6c7b6a62db075 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9432d0f6c60491b8a20b8dcf3561c84" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39607 } } peers { permanent_uuid: "2ebc7147b38d42c183f6c7b6a62db075" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46789 } } peers { permanent_uuid: "410592fd61c042c9a956aada0e560a12" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45795 } }
I20260619 16:44:04.149458 8444 heartbeater.cc:503] Master 127.0.97.62:39907 was elected leader, sending a full tablet report...
I20260619 16:44:04.150055 8470 ts_tablet_manager.cc:1428] T d1ea75f272c54ab3a32c2faeb373ec97 P 2ebc7147b38d42c183f6c7b6a62db075: Time spent starting tablet: real 0.014s user 0.013s sys 0.000s
I20260619 16:44:04.156306 8471 raft_consensus.cc:348] T d1ea75f272c54ab3a32c2faeb373ec97 P 410592fd61c042c9a956aada0e560a12 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9432d0f6c60491b8a20b8dcf3561c84" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39607 } } peers { permanent_uuid: "2ebc7147b38d42c183f6c7b6a62db075" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46789 } } peers { permanent_uuid: "410592fd61c042c9a956aada0e560a12" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45795 } }
I20260619 16:44:04.156651 8471 raft_consensus.cc:374] T d1ea75f272c54ab3a32c2faeb373ec97 P 410592fd61c042c9a956aada0e560a12 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:44:04.156762 8471 raft_consensus.cc:729] T d1ea75f272c54ab3a32c2faeb373ec97 P 410592fd61c042c9a956aada0e560a12 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 410592fd61c042c9a956aada0e560a12, State: Initialized, Role: FOLLOWER
I20260619 16:44:04.157330 8471 consensus_queue.cc:260] T d1ea75f272c54ab3a32c2faeb373ec97 P 410592fd61c042c9a956aada0e560a12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9432d0f6c60491b8a20b8dcf3561c84" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39607 } } peers { permanent_uuid: "2ebc7147b38d42c183f6c7b6a62db075" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46789 } } peers { permanent_uuid: "410592fd61c042c9a956aada0e560a12" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45795 } }
I20260619 16:44:04.158433 8305 heartbeater.cc:503] Master 127.0.97.62:39907 was elected leader, sending a full tablet report...
I20260619 16:44:04.158922 8471 ts_tablet_manager.cc:1428] T d1ea75f272c54ab3a32c2faeb373ec97 P 410592fd61c042c9a956aada0e560a12: Time spent starting tablet: real 0.011s user 0.009s sys 0.003s
I20260619 16:44:04.172328 8474 raft_consensus.cc:482] T d1ea75f272c54ab3a32c2faeb373ec97 P f9432d0f6c60491b8a20b8dcf3561c84 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 16:44:04.172609 8474 raft_consensus.cc:504] T d1ea75f272c54ab3a32c2faeb373ec97 P f9432d0f6c60491b8a20b8dcf3561c84 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9432d0f6c60491b8a20b8dcf3561c84" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39607 } } peers { permanent_uuid: "2ebc7147b38d42c183f6c7b6a62db075" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46789 } } peers { permanent_uuid: "410592fd61c042c9a956aada0e560a12" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45795 } }
I20260619 16:44:04.174443 8474 leader_election.cc:288] T d1ea75f272c54ab3a32c2faeb373ec97 P f9432d0f6c60491b8a20b8dcf3561c84 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2ebc7147b38d42c183f6c7b6a62db075 (127.0.97.3:46789), 410592fd61c042c9a956aada0e560a12 (127.0.97.2:45795)
I20260619 16:44:04.182758 8465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:44:04.175074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:51201 (local address 127.0.97.3:46789)
0619 16:44:04.175335 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0619 16:44:04.175340 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:44:04.175361 (+ 21us) server_negotiation.cc:407] Connection header received
0619 16:44:04.175456 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:44:04.175461 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:44:04.175530 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:44:04.175619 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:44:04.176393 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:44:04.177510 (+ 1117us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:44:04.180255 (+ 2745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:44:04.181517 (+ 1262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:44:04.181616 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:44:04.181726 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0619 16:44:04.182328 (+ 602us) server_negotiation.cc:299] Negotiation successful
0619 16:44:04.182471 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132}
I20260619 16:44:04.183449 8477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:44:04.174620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:46789 (local address 127.0.97.1:51201)
0619 16:44:04.175174 (+ 554us) negotiation.cc:107] Waiting for socket to connect
0619 16:44:04.175194 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 16:44:04.175335 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:44:04.175809 (+ 474us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:44:04.175815 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:44:04.175845 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:44:04.176225 (+ 380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:44:04.176237 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:44:04.177704 (+ 1467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:44:04.177711 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:44:04.180057 (+ 2346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:44:04.180074 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:44:04.181948 (+ 1874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:44:04.181955 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:44:04.182150 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:44:04.182184 (+ 34us) client_negotiation.cc:769] Sending connection context
0619 16:44:04.182957 (+ 773us) client_negotiation.cc:240] Negotiation successful
0619 16:44:04.183103 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":424,"thread_start_us":244,"threads_started":1}
I20260619 16:44:04.186892 8394 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1ea75f272c54ab3a32c2faeb373ec97" candidate_uuid: "f9432d0f6c60491b8a20b8dcf3561c84" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ebc7147b38d42c183f6c7b6a62db075" is_pre_election: true
I20260619 16:44:04.187397 8394 raft_consensus.cc:2454] T d1ea75f272c54ab3a32c2faeb373ec97 P 2ebc7147b38d42c183f6c7b6a62db075 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f9432d0f6c60491b8a20b8dcf3561c84 in term 0.
I20260619 16:44:04.188094 8048 leader_election.cc:302] T d1ea75f272c54ab3a32c2faeb373ec97 P f9432d0f6c60491b8a20b8dcf3561c84 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2ebc7147b38d42c183f6c7b6a62db075, f9432d0f6c60491b8a20b8dcf3561c84; no voters:
I20260619 16:44:04.188479 8474 raft_consensus.cc:2781] T d1ea75f272c54ab3a32c2faeb373ec97 P f9432d0f6c60491b8a20b8dcf3561c84 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 16:44:04.188596 8474 raft_consensus.cc:482] T d1ea75f272c54ab3a32c2faeb373ec97 P f9432d0f6c60491b8a20b8dcf3561c84 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 16:44:04.188683 8474 raft_consensus.cc:3037] T d1ea75f272c54ab3a32c2faeb373ec97 P f9432d0f6c60491b8a20b8dcf3561c84 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:44:04.190551 8467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:44:04.175532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:34217 (local address 127.0.97.2:45795)
0619 16:44:04.175704 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0619 16:44:04.175710 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:44:04.183768 (+ 8058us) server_negotiation.cc:407] Connection header received
0619 16:44:04.183935 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:44:04.183940 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:44:04.184019 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:44:04.184117 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:44:04.184479 (+ 362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:44:04.185643 (+ 1164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:44:04.187384 (+ 1741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:44:04.188322 (+ 938us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:44:04.188430 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:44:04.188505 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0619 16:44:04.190237 (+ 1732us) server_negotiation.cc:299] Negotiation successful
0619 16:44:04.190362 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260619 16:44:04.190596 8477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:44:04.175202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:45795 (local address 127.0.97.1:34217)
0619 16:44:04.183674 (+ 8472us) negotiation.cc:107] Waiting for socket to connect
0619 16:44:04.183693 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 16:44:04.183818 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:44:04.184131 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:44:04.184136 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:44:04.184158 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:44:04.184316 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:44:04.184326 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:44:04.185858 (+ 1532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:44:04.185864 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:44:04.187234 (+ 1370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:44:04.187249 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:44:04.189859 (+ 2610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:44:04.189866 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:44:04.190062 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:44:04.190093 (+ 31us) client_negotiation.cc:769] Sending connection context
0619 16:44:04.190190 (+ 97us) client_negotiation.cc:240] Negotiation successful
0619 16:44:04.190333 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8360,"thread_start_us":10726,"threads_started":1}
I20260619 16:44:04.191216 8474 raft_consensus.cc:504] T d1ea75f272c54ab3a32c2faeb373ec97 P f9432d0f6c60491b8a20b8dcf3561c84 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9432d0f6c60491b8a20b8dcf3561c84" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39607 } } peers { permanent_uuid: "2ebc7147b38d42c183f6c7b6a62db075" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46789 } } peers { permanent_uuid: "410592fd61c042c9a956aada0e560a12" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45795 } }
I20260619 16:44:04.191484 8255 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1ea75f272c54ab3a32c2faeb373ec97" candidate_uuid: "f9432d0f6c60491b8a20b8dcf3561c84" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "410592fd61c042c9a956aada0e560a12" is_pre_election: true
I20260619 16:44:04.191767 8474 leader_election.cc:288] T d1ea75f272c54ab3a32c2faeb373ec97 P f9432d0f6c60491b8a20b8dcf3561c84 [CANDIDATE]: Term 1 election: Requested vote from peers 2ebc7147b38d42c183f6c7b6a62db075 (127.0.97.3:46789), 410592fd61c042c9a956aada0e560a12 (127.0.97.2:45795)
I20260619 16:44:04.191996 8255 raft_consensus.cc:2454] T d1ea75f272c54ab3a32c2faeb373ec97 P 410592fd61c042c9a956aada0e560a12 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f9432d0f6c60491b8a20b8dcf3561c84 in term 0.
I20260619 16:44:04.192196 8394 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1ea75f272c54ab3a32c2faeb373ec97" candidate_uuid: "f9432d0f6c60491b8a20b8dcf3561c84" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ebc7147b38d42c183f6c7b6a62db075"
I20260619 16:44:04.192370 8394 raft_consensus.cc:3037] T d1ea75f272c54ab3a32c2faeb373ec97 P 2ebc7147b38d42c183f6c7b6a62db075 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:44:04.194504 8394 raft_consensus.cc:2454] T d1ea75f272c54ab3a32c2faeb373ec97 P 2ebc7147b38d42c183f6c7b6a62db075 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f9432d0f6c60491b8a20b8dcf3561c84 in term 1.
I20260619 16:44:04.194840 8048 leader_election.cc:302] T d1ea75f272c54ab3a32c2faeb373ec97 P f9432d0f6c60491b8a20b8dcf3561c84 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2ebc7147b38d42c183f6c7b6a62db075, f9432d0f6c60491b8a20b8dcf3561c84; no voters:
I20260619 16:44:04.194921 8254 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1ea75f272c54ab3a32c2faeb373ec97" candidate_uuid: "f9432d0f6c60491b8a20b8dcf3561c84" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "410592fd61c042c9a956aada0e560a12"
I20260619 16:44:04.195072 8474 raft_consensus.cc:2781] T d1ea75f272c54ab3a32c2faeb373ec97 P f9432d0f6c60491b8a20b8dcf3561c84 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:44:04.195163 8254 raft_consensus.cc:3037] T d1ea75f272c54ab3a32c2faeb373ec97 P 410592fd61c042c9a956aada0e560a12 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:44:04.197769 8254 raft_consensus.cc:2454] T d1ea75f272c54ab3a32c2faeb373ec97 P 410592fd61c042c9a956aada0e560a12 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f9432d0f6c60491b8a20b8dcf3561c84 in term 1.
I20260619 16:44:04.197988 8474 raft_consensus.cc:686] T d1ea75f272c54ab3a32c2faeb373ec97 P f9432d0f6c60491b8a20b8dcf3561c84 [term 1 LEADER]: Becoming Leader. State: Replica: f9432d0f6c60491b8a20b8dcf3561c84, State: Running, Role: LEADER
I20260619 16:44:04.198552 8474 consensus_queue.cc:237] T d1ea75f272c54ab3a32c2faeb373ec97 P f9432d0f6c60491b8a20b8dcf3561c84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9432d0f6c60491b8a20b8dcf3561c84" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39607 } } peers { permanent_uuid: "2ebc7147b38d42c183f6c7b6a62db075" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46789 } } peers { permanent_uuid: "410592fd61c042c9a956aada0e560a12" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45795 } }
I20260619 16:44:04.202936 7958 catalog_manager.cc:5495] T d1ea75f272c54ab3a32c2faeb373ec97 P f9432d0f6c60491b8a20b8dcf3561c84 reported cstate change: term changed from 0 to 1, leader changed from <none> to f9432d0f6c60491b8a20b8dcf3561c84 (127.0.97.1). New cstate: current_term: 1 leader_uuid: "f9432d0f6c60491b8a20b8dcf3561c84" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9432d0f6c60491b8a20b8dcf3561c84" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39607 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2ebc7147b38d42c183f6c7b6a62db075" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46789 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "410592fd61c042c9a956aada0e560a12" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45795 } health_report { overall_health: UNKNOWN } } }
I20260619 16:44:04.231841 8463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:44:04.227060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40888 (local address 127.0.97.1:39607)
0619 16:44:04.227211 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0619 16:44:04.227216 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:44:04.227268 (+ 52us) server_negotiation.cc:407] Connection header received
0619 16:44:04.227458 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:44:04.227462 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:44:04.227541 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:44:04.227656 (+ 115us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 16:44:04.228023 (+ 367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:44:04.228703 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:44:04.230106 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:44:04.230481 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:44:04.230575 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:44:04.230937 (+ 362us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 16:44:04.231418 (+ 481us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 16:44:04.231515 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0619 16:44:04.231592 (+ 77us) server_negotiation.cc:299] Negotiation successful
0619 16:44:04.231652 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260619 16:44:04.239271 8394 raft_consensus.cc:1261] T d1ea75f272c54ab3a32c2faeb373ec97 P 2ebc7147b38d42c183f6c7b6a62db075 [term 1 FOLLOWER]: Refusing update from remote peer f9432d0f6c60491b8a20b8dcf3561c84: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:44:04.239264 8254 raft_consensus.cc:1261] T d1ea75f272c54ab3a32c2faeb373ec97 P 410592fd61c042c9a956aada0e560a12 [term 1 FOLLOWER]: Refusing update from remote peer f9432d0f6c60491b8a20b8dcf3561c84: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:44:04.240115 8479 consensus_queue.cc:1035] T d1ea75f272c54ab3a32c2faeb373ec97 P f9432d0f6c60491b8a20b8dcf3561c84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ebc7147b38d42c183f6c7b6a62db075" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:44:04.240464 8474 consensus_queue.cc:1035] T d1ea75f272c54ab3a32c2faeb373ec97 P f9432d0f6c60491b8a20b8dcf3561c84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "410592fd61c042c9a956aada0e560a12" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260619 16:44:04.245543 8445 tablet.cc:2369] T d1ea75f272c54ab3a32c2faeb373ec97 P 2ebc7147b38d42c183f6c7b6a62db075: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 16:44:04.251803 8483 mvcc.cc:205] Tried to move back new op lower bound from 7298610971595464704 to 7298610971440672768. Current Snapshot: MvccSnapshot[applied={T|T < 7298610971595464704}]
I20260619 16:44:04.264995 7958 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48018:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:44:04.265231 7958 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48018:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:44:04.268303 7958 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ed5a208fb1504c2dbce0eceab08ad43c: Sending DeleteTablet for 3 replicas of tablet d1ea75f272c54ab3a32c2faeb373ec97
I20260619 16:44:04.269394 8096 tablet_service.cc:1514] Processing DeleteTablet for tablet d1ea75f272c54ab3a32c2faeb373ec97 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:44:04 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:40886
I20260619 16:44:04.269577 8235 tablet_service.cc:1514] Processing DeleteTablet for tablet d1ea75f272c54ab3a32c2faeb373ec97 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:44:04 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:59360
I20260619 16:44:04.269573 8374 tablet_service.cc:1514] Processing DeleteTablet for tablet d1ea75f272c54ab3a32c2faeb373ec97 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:44:04 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:60330
I20260619 16:44:04.273758 7958 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48018:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 16:44:04.275216 7958 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 16:44:04.278241 8492 tablet_replica.cc:330] T d1ea75f272c54ab3a32c2faeb373ec97 P f9432d0f6c60491b8a20b8dcf3561c84: stopping tablet replica
I20260619 16:44:04.278831 8492 raft_consensus.cc:2229] T d1ea75f272c54ab3a32c2faeb373ec97 P f9432d0f6c60491b8a20b8dcf3561c84 [term 1 LEADER]: Raft consensus shutting down.
I20260619 16:44:04.279665 8492 raft_consensus.cc:2258] T d1ea75f272c54ab3a32c2faeb373ec97 P f9432d0f6c60491b8a20b8dcf3561c84 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:44:04.281940 8492 ts_tablet_manager.cc:1905] T d1ea75f272c54ab3a32c2faeb373ec97 P f9432d0f6c60491b8a20b8dcf3561c84: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:44:04.282897 8494 tablet_replica.cc:330] T d1ea75f272c54ab3a32c2faeb373ec97 P 410592fd61c042c9a956aada0e560a12: stopping tablet replica
I20260619 16:44:04.283396 8494 raft_consensus.cc:2229] T d1ea75f272c54ab3a32c2faeb373ec97 P 410592fd61c042c9a956aada0e560a12 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 16:44:04.284355 8494 raft_consensus.cc:2258] T d1ea75f272c54ab3a32c2faeb373ec97 P 410592fd61c042c9a956aada0e560a12 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:44:04.285763 8096 tablet_service.cc:1467] Processing CreateTablet for tablet 20e0d115a7d04b30b0107038155568ee (DEFAULT_TABLE table=test-table [id=d6515c8cd5554ba4ad1055ea387c6d93]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:44:04.285951 8373 tablet_service.cc:1467] Processing CreateTablet for tablet 20e0d115a7d04b30b0107038155568ee (DEFAULT_TABLE table=test-table [id=d6515c8cd5554ba4ad1055ea387c6d93]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:44:04.286206 8096 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 20e0d115a7d04b30b0107038155568ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:44:04.286516 8373 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 20e0d115a7d04b30b0107038155568ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:44:04.287518 8493 tablet_replica.cc:330] T d1ea75f272c54ab3a32c2faeb373ec97 P 2ebc7147b38d42c183f6c7b6a62db075: stopping tablet replica
I20260619 16:44:04.288335 8494 ts_tablet_manager.cc:1905] T d1ea75f272c54ab3a32c2faeb373ec97 P 410592fd61c042c9a956aada0e560a12: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:44:04.288798 8493 raft_consensus.cc:2229] T d1ea75f272c54ab3a32c2faeb373ec97 P 2ebc7147b38d42c183f6c7b6a62db075 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 16:44:04.289299 8493 raft_consensus.cc:2258] T d1ea75f272c54ab3a32c2faeb373ec97 P 2ebc7147b38d42c183f6c7b6a62db075 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:44:04.292752 8235 tablet_service.cc:1467] Processing CreateTablet for tablet 20e0d115a7d04b30b0107038155568ee (DEFAULT_TABLE table=test-table [id=d6515c8cd5554ba4ad1055ea387c6d93]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:44:04.293043 8470 tablet_bootstrap.cc:492] T 20e0d115a7d04b30b0107038155568ee P 2ebc7147b38d42c183f6c7b6a62db075: Bootstrap starting.
I20260619 16:44:04.293188 8235 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 20e0d115a7d04b30b0107038155568ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:44:04.294538 8470 tablet_bootstrap.cc:654] T 20e0d115a7d04b30b0107038155568ee P 2ebc7147b38d42c183f6c7b6a62db075: Neither blocks nor log segments found. Creating new log.
I20260619 16:44:04.294960 8494 ts_tablet_manager.cc:1918] T d1ea75f272c54ab3a32c2faeb373ec97 P 410592fd61c042c9a956aada0e560a12: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 16:44:04.295127 8494 log.cc:1196] T d1ea75f272c54ab3a32c2faeb373ec97 P 410592fd61c042c9a956aada0e560a12: Deleting WAL directory at /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887385903558-388-0/minicluster-data/ts-1/wal/wals/d1ea75f272c54ab3a32c2faeb373ec97
I20260619 16:44:04.295624 8494 ts_tablet_manager.cc:1939] T d1ea75f272c54ab3a32c2faeb373ec97 P 410592fd61c042c9a956aada0e560a12: Deleting consensus metadata
I20260619 16:44:04.296074 8468 tablet_bootstrap.cc:492] T 20e0d115a7d04b30b0107038155568ee P f9432d0f6c60491b8a20b8dcf3561c84: Bootstrap starting.
I20260619 16:44:04.297647 8468 tablet_bootstrap.cc:654] T 20e0d115a7d04b30b0107038155568ee P f9432d0f6c60491b8a20b8dcf3561c84: Neither blocks nor log segments found. Creating new log.
I20260619 16:44:04.298156 7945 catalog_manager.cc:4841] TS 410592fd61c042c9a956aada0e560a12 (127.0.97.2:45795): tablet d1ea75f272c54ab3a32c2faeb373ec97 (table test-table [id=0daaa89c199e4b7abd1cf1dfb7caa624]) successfully deleted
I20260619 16:44:04.299283 8492 ts_tablet_manager.cc:1918] T d1ea75f272c54ab3a32c2faeb373ec97 P f9432d0f6c60491b8a20b8dcf3561c84: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 16:44:04.299451 8492 log.cc:1196] T d1ea75f272c54ab3a32c2faeb373ec97 P f9432d0f6c60491b8a20b8dcf3561c84: Deleting WAL directory at /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887385903558-388-0/minicluster-data/ts-0/wal/wals/d1ea75f272c54ab3a32c2faeb373ec97
I20260619 16:44:04.299950 8492 ts_tablet_manager.cc:1939] T d1ea75f272c54ab3a32c2faeb373ec97 P f9432d0f6c60491b8a20b8dcf3561c84: Deleting consensus metadata
I20260619 16:44:04.300426 8470 tablet_bootstrap.cc:492] T 20e0d115a7d04b30b0107038155568ee P 2ebc7147b38d42c183f6c7b6a62db075: No bootstrap required, opened a new log
I20260619 16:44:04.300601 8470 ts_tablet_manager.cc:1397] T 20e0d115a7d04b30b0107038155568ee P 2ebc7147b38d42c183f6c7b6a62db075: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.000s
I20260619 16:44:04.300711 8493 ts_tablet_manager.cc:1905] T d1ea75f272c54ab3a32c2faeb373ec97 P 2ebc7147b38d42c183f6c7b6a62db075: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:44:04.301453 7943 catalog_manager.cc:4841] TS f9432d0f6c60491b8a20b8dcf3561c84 (127.0.97.1:39607): tablet d1ea75f272c54ab3a32c2faeb373ec97 (table test-table [id=0daaa89c199e4b7abd1cf1dfb7caa624]) successfully deleted
I20260619 16:44:04.301630 8470 raft_consensus.cc:348] T 20e0d115a7d04b30b0107038155568ee P 2ebc7147b38d42c183f6c7b6a62db075 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9432d0f6c60491b8a20b8dcf3561c84" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39607 } } peers { permanent_uuid: "410592fd61c042c9a956aada0e560a12" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45795 } } peers { permanent_uuid: "2ebc7147b38d42c183f6c7b6a62db075" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46789 } }
I20260619 16:44:04.302345 8470 raft_consensus.cc:374] T 20e0d115a7d04b30b0107038155568ee P 2ebc7147b38d42c183f6c7b6a62db075 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:44:04.302472 8470 raft_consensus.cc:729] T 20e0d115a7d04b30b0107038155568ee P 2ebc7147b38d42c183f6c7b6a62db075 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ebc7147b38d42c183f6c7b6a62db075, State: Initialized, Role: FOLLOWER
I20260619 16:44:04.303723 8470 consensus_queue.cc:260] T 20e0d115a7d04b30b0107038155568ee P 2ebc7147b38d42c183f6c7b6a62db075 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9432d0f6c60491b8a20b8dcf3561c84" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39607 } } peers { permanent_uuid: "410592fd61c042c9a956aada0e560a12" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45795 } } peers { permanent_uuid: "2ebc7147b38d42c183f6c7b6a62db075" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46789 } }
I20260619 16:44:04.304721 8471 tablet_bootstrap.cc:492] T 20e0d115a7d04b30b0107038155568ee P 410592fd61c042c9a956aada0e560a12: Bootstrap starting.
I20260619 16:44:04.305087 8470 ts_tablet_manager.cc:1428] T 20e0d115a7d04b30b0107038155568ee P 2ebc7147b38d42c183f6c7b6a62db075: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260619 16:44:04.306255 8471 tablet_bootstrap.cc:654] T 20e0d115a7d04b30b0107038155568ee P 410592fd61c042c9a956aada0e560a12: Neither blocks nor log segments found. Creating new log.
I20260619 16:44:04.311028 8471 tablet_bootstrap.cc:492] T 20e0d115a7d04b30b0107038155568ee P 410592fd61c042c9a956aada0e560a12: No bootstrap required, opened a new log
I20260619 16:44:04.311074 8468 tablet_bootstrap.cc:492] T 20e0d115a7d04b30b0107038155568ee P f9432d0f6c60491b8a20b8dcf3561c84: No bootstrap required, opened a new log
I20260619 16:44:04.311170 8471 ts_tablet_manager.cc:1397] T 20e0d115a7d04b30b0107038155568ee P 410592fd61c042c9a956aada0e560a12: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260619 16:44:04.311208 8468 ts_tablet_manager.cc:1397] T 20e0d115a7d04b30b0107038155568ee P f9432d0f6c60491b8a20b8dcf3561c84: Time spent bootstrapping tablet: real 0.015s user 0.003s sys 0.004s
I20260619 16:44:04.311945 8471 raft_consensus.cc:348] T 20e0d115a7d04b30b0107038155568ee P 410592fd61c042c9a956aada0e560a12 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9432d0f6c60491b8a20b8dcf3561c84" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39607 } } peers { permanent_uuid: "410592fd61c042c9a956aada0e560a12" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45795 } } peers { permanent_uuid: "2ebc7147b38d42c183f6c7b6a62db075" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46789 } }
I20260619 16:44:04.311980 8468 raft_consensus.cc:348] T 20e0d115a7d04b30b0107038155568ee P f9432d0f6c60491b8a20b8dcf3561c84 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9432d0f6c60491b8a20b8dcf3561c84" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39607 } } peers { permanent_uuid: "410592fd61c042c9a956aada0e560a12" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45795 } } peers { permanent_uuid: "2ebc7147b38d42c183f6c7b6a62db075" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46789 } }
I20260619 16:44:04.312151 8471 raft_consensus.cc:374] T 20e0d115a7d04b30b0107038155568ee P 410592fd61c042c9a956aada0e560a12 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:44:04.312192 8468 raft_consensus.cc:374] T 20e0d115a7d04b30b0107038155568ee P f9432d0f6c60491b8a20b8dcf3561c84 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:44:04.312242 8471 raft_consensus.cc:729] T 20e0d115a7d04b30b0107038155568ee P 410592fd61c042c9a956aada0e560a12 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 410592fd61c042c9a956aada0e560a12, State: Initialized, Role: FOLLOWER
I20260619 16:44:04.312270 8468 raft_consensus.cc:729] T 20e0d115a7d04b30b0107038155568ee P f9432d0f6c60491b8a20b8dcf3561c84 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f9432d0f6c60491b8a20b8dcf3561c84, State: Initialized, Role: FOLLOWER
I20260619 16:44:04.312454 8471 consensus_queue.cc:260] T 20e0d115a7d04b30b0107038155568ee P 410592fd61c042c9a956aada0e560a12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9432d0f6c60491b8a20b8dcf3561c84" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39607 } } peers { permanent_uuid: "410592fd61c042c9a956aada0e560a12" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45795 } } peers { permanent_uuid: "2ebc7147b38d42c183f6c7b6a62db075" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46789 } }
I20260619 16:44:04.312541 8468 consensus_queue.cc:260] T 20e0d115a7d04b30b0107038155568ee P f9432d0f6c60491b8a20b8dcf3561c84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9432d0f6c60491b8a20b8dcf3561c84" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39607 } } peers { permanent_uuid: "410592fd61c042c9a956aada0e560a12" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45795 } } peers { permanent_uuid: "2ebc7147b38d42c183f6c7b6a62db075" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46789 } }
I20260619 16:44:04.312948 8471 ts_tablet_manager.cc:1428] T 20e0d115a7d04b30b0107038155568ee P 410592fd61c042c9a956aada0e560a12: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260619 16:44:04.313095 8493 ts_tablet_manager.cc:1918] T d1ea75f272c54ab3a32c2faeb373ec97 P 2ebc7147b38d42c183f6c7b6a62db075: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 16:44:04.313239 8493 log.cc:1196] T d1ea75f272c54ab3a32c2faeb373ec97 P 2ebc7147b38d42c183f6c7b6a62db075: Deleting WAL directory at /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887385903558-388-0/minicluster-data/ts-2/wal/wals/d1ea75f272c54ab3a32c2faeb373ec97
I20260619 16:44:04.313721 8493 ts_tablet_manager.cc:1939] T d1ea75f272c54ab3a32c2faeb373ec97 P 2ebc7147b38d42c183f6c7b6a62db075: Deleting consensus metadata
I20260619 16:44:04.315529 7945 catalog_manager.cc:4841] TS 2ebc7147b38d42c183f6c7b6a62db075 (127.0.97.3:46789): tablet d1ea75f272c54ab3a32c2faeb373ec97 (table test-table [id=0daaa89c199e4b7abd1cf1dfb7caa624]) successfully deleted
I20260619 16:44:04.315618 8468 ts_tablet_manager.cc:1428] T 20e0d115a7d04b30b0107038155568ee P f9432d0f6c60491b8a20b8dcf3561c84: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
W20260619 16:44:04.324141 8167 tablet.cc:2369] T 20e0d115a7d04b30b0107038155568ee P f9432d0f6c60491b8a20b8dcf3561c84: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 16:44:04.540601 8306 tablet.cc:2369] T 20e0d115a7d04b30b0107038155568ee P 410592fd61c042c9a956aada0e560a12: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 16:44:04.580289 8474 raft_consensus.cc:482] T 20e0d115a7d04b30b0107038155568ee P f9432d0f6c60491b8a20b8dcf3561c84 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 16:44:04.580533 8474 raft_consensus.cc:504] T 20e0d115a7d04b30b0107038155568ee P f9432d0f6c60491b8a20b8dcf3561c84 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9432d0f6c60491b8a20b8dcf3561c84" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39607 } } peers { permanent_uuid: "410592fd61c042c9a956aada0e560a12" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45795 } } peers { permanent_uuid: "2ebc7147b38d42c183f6c7b6a62db075" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46789 } }
I20260619 16:44:04.581346 8474 leader_election.cc:288] T 20e0d115a7d04b30b0107038155568ee P f9432d0f6c60491b8a20b8dcf3561c84 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 410592fd61c042c9a956aada0e560a12 (127.0.97.2:45795), 2ebc7147b38d42c183f6c7b6a62db075 (127.0.97.3:46789)
I20260619 16:44:04.581884 8394 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20e0d115a7d04b30b0107038155568ee" candidate_uuid: "f9432d0f6c60491b8a20b8dcf3561c84" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ebc7147b38d42c183f6c7b6a62db075" is_pre_election: true
I20260619 16:44:04.581887 8254 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20e0d115a7d04b30b0107038155568ee" candidate_uuid: "f9432d0f6c60491b8a20b8dcf3561c84" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "410592fd61c042c9a956aada0e560a12" is_pre_election: true
I20260619 16:44:04.582122 8254 raft_consensus.cc:2454] T 20e0d115a7d04b30b0107038155568ee P 410592fd61c042c9a956aada0e560a12 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f9432d0f6c60491b8a20b8dcf3561c84 in term 0.
I20260619 16:44:04.582139 8394 raft_consensus.cc:2454] T 20e0d115a7d04b30b0107038155568ee P 2ebc7147b38d42c183f6c7b6a62db075 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f9432d0f6c60491b8a20b8dcf3561c84 in term 0.
I20260619 16:44:04.582563 8048 leader_election.cc:302] T 20e0d115a7d04b30b0107038155568ee P f9432d0f6c60491b8a20b8dcf3561c84 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 410592fd61c042c9a956aada0e560a12, f9432d0f6c60491b8a20b8dcf3561c84; no voters:
I20260619 16:44:04.582839 8474 raft_consensus.cc:2781] T 20e0d115a7d04b30b0107038155568ee P f9432d0f6c60491b8a20b8dcf3561c84 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 16:44:04.582952 8474 raft_consensus.cc:482] T 20e0d115a7d04b30b0107038155568ee P f9432d0f6c60491b8a20b8dcf3561c84 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 16:44:04.583046 8474 raft_consensus.cc:3037] T 20e0d115a7d04b30b0107038155568ee P f9432d0f6c60491b8a20b8dcf3561c84 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:44:04.585116 8474 raft_consensus.cc:504] T 20e0d115a7d04b30b0107038155568ee P f9432d0f6c60491b8a20b8dcf3561c84 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9432d0f6c60491b8a20b8dcf3561c84" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39607 } } peers { permanent_uuid: "410592fd61c042c9a956aada0e560a12" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45795 } } peers { permanent_uuid: "2ebc7147b38d42c183f6c7b6a62db075" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46789 } }
I20260619 16:44:04.585628 8474 leader_election.cc:288] T 20e0d115a7d04b30b0107038155568ee P f9432d0f6c60491b8a20b8dcf3561c84 [CANDIDATE]: Term 1 election: Requested vote from peers 410592fd61c042c9a956aada0e560a12 (127.0.97.2:45795), 2ebc7147b38d42c183f6c7b6a62db075 (127.0.97.3:46789)
I20260619 16:44:04.586172 8394 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20e0d115a7d04b30b0107038155568ee" candidate_uuid: "f9432d0f6c60491b8a20b8dcf3561c84" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ebc7147b38d42c183f6c7b6a62db075"
I20260619 16:44:04.586171 8254 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20e0d115a7d04b30b0107038155568ee" candidate_uuid: "f9432d0f6c60491b8a20b8dcf3561c84" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "410592fd61c042c9a956aada0e560a12"
I20260619 16:44:04.586331 8394 raft_consensus.cc:3037] T 20e0d115a7d04b30b0107038155568ee P 2ebc7147b38d42c183f6c7b6a62db075 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:44:04.586365 8254 raft_consensus.cc:3037] T 20e0d115a7d04b30b0107038155568ee P 410592fd61c042c9a956aada0e560a12 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:44:04.588265 8394 raft_consensus.cc:2454] T 20e0d115a7d04b30b0107038155568ee P 2ebc7147b38d42c183f6c7b6a62db075 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f9432d0f6c60491b8a20b8dcf3561c84 in term 1.
I20260619 16:44:04.589242 8254 raft_consensus.cc:2454] T 20e0d115a7d04b30b0107038155568ee P 410592fd61c042c9a956aada0e560a12 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f9432d0f6c60491b8a20b8dcf3561c84 in term 1.
I20260619 16:44:04.589490 8048 leader_election.cc:302] T 20e0d115a7d04b30b0107038155568ee P f9432d0f6c60491b8a20b8dcf3561c84 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2ebc7147b38d42c183f6c7b6a62db075, f9432d0f6c60491b8a20b8dcf3561c84; no voters:
I20260619 16:44:04.589787 8474 raft_consensus.cc:2781] T 20e0d115a7d04b30b0107038155568ee P f9432d0f6c60491b8a20b8dcf3561c84 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:44:04.589972 8474 raft_consensus.cc:686] T 20e0d115a7d04b30b0107038155568ee P f9432d0f6c60491b8a20b8dcf3561c84 [term 1 LEADER]: Becoming Leader. State: Replica: f9432d0f6c60491b8a20b8dcf3561c84, State: Running, Role: LEADER
I20260619 16:44:04.590252 8474 consensus_queue.cc:237] T 20e0d115a7d04b30b0107038155568ee P f9432d0f6c60491b8a20b8dcf3561c84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9432d0f6c60491b8a20b8dcf3561c84" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39607 } } peers { permanent_uuid: "410592fd61c042c9a956aada0e560a12" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45795 } } peers { permanent_uuid: "2ebc7147b38d42c183f6c7b6a62db075" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46789 } }
I20260619 16:44:04.592707 7959 catalog_manager.cc:5495] T 20e0d115a7d04b30b0107038155568ee P f9432d0f6c60491b8a20b8dcf3561c84 reported cstate change: term changed from 0 to 1, leader changed from <none> to f9432d0f6c60491b8a20b8dcf3561c84 (127.0.97.1). New cstate: current_term: 1 leader_uuid: "f9432d0f6c60491b8a20b8dcf3561c84" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9432d0f6c60491b8a20b8dcf3561c84" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39607 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "410592fd61c042c9a956aada0e560a12" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45795 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2ebc7147b38d42c183f6c7b6a62db075" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46789 } health_report { overall_health: UNKNOWN } } }
Jun 19 16:44:04 dist-test-slave-wvz7 krb5kdc[7914](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887442, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.62@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260619 16:44:04.692376 8505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:44:04.679649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39907 (local address 127.0.0.1:48026)
0619 16:44:04.680335 (+ 686us) negotiation.cc:107] Waiting for socket to connect
0619 16:44:04.680356 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 16:44:04.680499 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:44:04.680829 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:44:04.680834 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:44:04.681511 (+ 677us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:44:04.681767 (+ 256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:44:04.681782 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:44:04.682844 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:44:04.682851 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:44:04.683884 (+ 1033us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:44:04.683898 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:44:04.684575 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:44:04.684581 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:44:04.684769 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:44:04.685377 (+ 608us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:44:04.685404 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:44:04.687551 (+ 2147us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:44:04.690148 (+ 2597us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:44:04.690157 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:44:04.690162 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:44:04.690543 (+ 381us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:44:04.690862 (+ 319us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:44:04.690867 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:44:04.690871 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:44:04.691053 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:44:04.691516 (+ 463us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:44:04.691523 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:44:04.691699 (+ 176us) client_negotiation.cc:769] Sending connection context
0619 16:44:04.691859 (+ 160us) client_negotiation.cc:240] Negotiation successful
0619 16:44:04.692052 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":547,"thread_start_us":376,"threads_started":1}
I20260619 16:44:04.692420 8504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:44:04.679659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48026 (local address 127.0.97.62:39907)
0619 16:44:04.680308 (+ 649us) server_negotiation.cc:206] Beginning negotiation
0619 16:44:04.680315 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:44:04.680432 (+ 117us) server_negotiation.cc:407] Connection header received
0619 16:44:04.680602 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:44:04.680608 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:44:04.680674 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:44:04.680800 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:44:04.681901 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:44:04.682694 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:44:04.684020 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:44:04.684427 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:44:04.684542 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:44:04.687735 (+ 3193us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:44:04.687742 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:44:04.687746 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:44:04.687781 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:44:04.690022 (+ 2241us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:44:04.690639 (+ 617us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:44:04.690643 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:44:04.690645 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:44:04.690741 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:44:04.691160 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:44:04.691163 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:44:04.691165 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:44:04.691402 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:44:04.691504 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0619 16:44:04.691937 (+ 433us) server_negotiation.cc:299] Negotiation successful
0619 16:44:04.692155 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":65,"server-negotiator.queue_time_us":487,"thread_start_us":290,"threads_started":1}
I20260619 16:44:04.696859 7959 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:48026:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260619 16:44:04.697829 7959 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 16:44:04.703325 8096 tablet_service.cc:1467] Processing CreateTablet for tablet f210589bda404913a7f456f39163a60d (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=7c47bee65e2a4aa6be1229f4ca2cae46]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 16:44:04.703397 8235 tablet_service.cc:1467] Processing CreateTablet for tablet f210589bda404913a7f456f39163a60d (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=7c47bee65e2a4aa6be1229f4ca2cae46]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 16:44:04.703567 8373 tablet_service.cc:1467] Processing CreateTablet for tablet f210589bda404913a7f456f39163a60d (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=7c47bee65e2a4aa6be1229f4ca2cae46]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 16:44:04.703805 8096 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f210589bda404913a7f456f39163a60d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:44:04.703902 8235 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f210589bda404913a7f456f39163a60d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:44:04.704017 8373 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f210589bda404913a7f456f39163a60d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:44:04.708350 8468 tablet_bootstrap.cc:492] T f210589bda404913a7f456f39163a60d P f9432d0f6c60491b8a20b8dcf3561c84: Bootstrap starting.
I20260619 16:44:04.709795 8468 tablet_bootstrap.cc:654] T f210589bda404913a7f456f39163a60d P f9432d0f6c60491b8a20b8dcf3561c84: Neither blocks nor log segments found. Creating new log.
I20260619 16:44:04.710498 8470 tablet_bootstrap.cc:492] T f210589bda404913a7f456f39163a60d P 2ebc7147b38d42c183f6c7b6a62db075: Bootstrap starting.
I20260619 16:44:04.710592 8471 tablet_bootstrap.cc:492] T f210589bda404913a7f456f39163a60d P 410592fd61c042c9a956aada0e560a12: Bootstrap starting.
I20260619 16:44:04.711946 8471 tablet_bootstrap.cc:654] T f210589bda404913a7f456f39163a60d P 410592fd61c042c9a956aada0e560a12: Neither blocks nor log segments found. Creating new log.
I20260619 16:44:04.712131 8470 tablet_bootstrap.cc:654] T f210589bda404913a7f456f39163a60d P 2ebc7147b38d42c183f6c7b6a62db075: Neither blocks nor log segments found. Creating new log.
I20260619 16:44:04.713629 8471 tablet_bootstrap.cc:492] T f210589bda404913a7f456f39163a60d P 410592fd61c042c9a956aada0e560a12: No bootstrap required, opened a new log
I20260619 16:44:04.713783 8471 ts_tablet_manager.cc:1397] T f210589bda404913a7f456f39163a60d P 410592fd61c042c9a956aada0e560a12: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.001s
I20260619 16:44:04.714457 8471 raft_consensus.cc:348] T f210589bda404913a7f456f39163a60d P 410592fd61c042c9a956aada0e560a12 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9432d0f6c60491b8a20b8dcf3561c84" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39607 } } peers { permanent_uuid: "410592fd61c042c9a956aada0e560a12" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45795 } } peers { permanent_uuid: "2ebc7147b38d42c183f6c7b6a62db075" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46789 } }
I20260619 16:44:04.714701 8471 raft_consensus.cc:374] T f210589bda404913a7f456f39163a60d P 410592fd61c042c9a956aada0e560a12 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:44:04.714775 8471 raft_consensus.cc:729] T f210589bda404913a7f456f39163a60d P 410592fd61c042c9a956aada0e560a12 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 410592fd61c042c9a956aada0e560a12, State: Initialized, Role: FOLLOWER
I20260619 16:44:04.715008 8471 consensus_queue.cc:260] T f210589bda404913a7f456f39163a60d P 410592fd61c042c9a956aada0e560a12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9432d0f6c60491b8a20b8dcf3561c84" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39607 } } peers { permanent_uuid: "410592fd61c042c9a956aada0e560a12" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45795 } } peers { permanent_uuid: "2ebc7147b38d42c183f6c7b6a62db075" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46789 } }
I20260619 16:44:04.715535 8471 ts_tablet_manager.cc:1428] T f210589bda404913a7f456f39163a60d P 410592fd61c042c9a956aada0e560a12: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260619 16:44:04.715492 8476 tablet_replica.cc:439] T f210589bda404913a7f456f39163a60d P 410592fd61c042c9a956aada0e560a12: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9432d0f6c60491b8a20b8dcf3561c84" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39607 } } peers { permanent_uuid: "410592fd61c042c9a956aada0e560a12" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45795 } } peers { permanent_uuid: "2ebc7147b38d42c183f6c7b6a62db075" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46789 } } }
I20260619 16:44:04.715862 8476 tablet_replica.cc:442] T f210589bda404913a7f456f39163a60d P 410592fd61c042c9a956aada0e560a12: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 16:44:04.717082 8468 tablet_bootstrap.cc:492] T f210589bda404913a7f456f39163a60d P f9432d0f6c60491b8a20b8dcf3561c84: No bootstrap required, opened a new log
I20260619 16:44:04.717231 8468 ts_tablet_manager.cc:1397] T f210589bda404913a7f456f39163a60d P f9432d0f6c60491b8a20b8dcf3561c84: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.004s
I20260619 16:44:04.717816 8470 tablet_bootstrap.cc:492] T f210589bda404913a7f456f39163a60d P 2ebc7147b38d42c183f6c7b6a62db075: No bootstrap required, opened a new log
I20260619 16:44:04.717808 8468 raft_consensus.cc:348] T f210589bda404913a7f456f39163a60d P f9432d0f6c60491b8a20b8dcf3561c84 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9432d0f6c60491b8a20b8dcf3561c84" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39607 } } peers { permanent_uuid: "410592fd61c042c9a956aada0e560a12" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45795 } } peers { permanent_uuid: "2ebc7147b38d42c183f6c7b6a62db075" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46789 } }
I20260619 16:44:04.717975 8470 ts_tablet_manager.cc:1397] T f210589bda404913a7f456f39163a60d P 2ebc7147b38d42c183f6c7b6a62db075: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.004s
I20260619 16:44:04.718039 8468 raft_consensus.cc:374] T f210589bda404913a7f456f39163a60d P f9432d0f6c60491b8a20b8dcf3561c84 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:44:04.718111 8468 raft_consensus.cc:729] T f210589bda404913a7f456f39163a60d P f9432d0f6c60491b8a20b8dcf3561c84 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f9432d0f6c60491b8a20b8dcf3561c84, State: Initialized, Role: FOLLOWER
I20260619 16:44:04.718364 8468 consensus_queue.cc:260] T f210589bda404913a7f456f39163a60d P f9432d0f6c60491b8a20b8dcf3561c84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9432d0f6c60491b8a20b8dcf3561c84" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39607 } } peers { permanent_uuid: "410592fd61c042c9a956aada0e560a12" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45795 } } peers { permanent_uuid: "2ebc7147b38d42c183f6c7b6a62db075" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46789 } }
I20260619 16:44:04.718673 8470 raft_consensus.cc:348] T f210589bda404913a7f456f39163a60d P 2ebc7147b38d42c183f6c7b6a62db075 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9432d0f6c60491b8a20b8dcf3561c84" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39607 } } peers { permanent_uuid: "410592fd61c042c9a956aada0e560a12" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45795 } } peers { permanent_uuid: "2ebc7147b38d42c183f6c7b6a62db075" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46789 } }
I20260619 16:44:04.718895 8470 raft_consensus.cc:374] T f210589bda404913a7f456f39163a60d P 2ebc7147b38d42c183f6c7b6a62db075 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:44:04.718770 8474 tablet_replica.cc:439] T f210589bda404913a7f456f39163a60d P f9432d0f6c60491b8a20b8dcf3561c84: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9432d0f6c60491b8a20b8dcf3561c84" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39607 } } peers { permanent_uuid: "410592fd61c042c9a956aada0e560a12" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45795 } } peers { permanent_uuid: "2ebc7147b38d42c183f6c7b6a62db075" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46789 } } }
I20260619 16:44:04.718969 8470 raft_consensus.cc:729] T f210589bda404913a7f456f39163a60d P 2ebc7147b38d42c183f6c7b6a62db075 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ebc7147b38d42c183f6c7b6a62db075, State: Initialized, Role: FOLLOWER
I20260619 16:44:04.719058 8474 tablet_replica.cc:442] T f210589bda404913a7f456f39163a60d P f9432d0f6c60491b8a20b8dcf3561c84: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 16:44:04.719338 8470 consensus_queue.cc:260] T f210589bda404913a7f456f39163a60d P 2ebc7147b38d42c183f6c7b6a62db075 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9432d0f6c60491b8a20b8dcf3561c84" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39607 } } peers { permanent_uuid: "410592fd61c042c9a956aada0e560a12" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45795 } } peers { permanent_uuid: "2ebc7147b38d42c183f6c7b6a62db075" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46789 } }
I20260619 16:44:04.719918 8470 ts_tablet_manager.cc:1428] T f210589bda404913a7f456f39163a60d P 2ebc7147b38d42c183f6c7b6a62db075: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260619 16:44:04.719776 8475 tablet_replica.cc:439] T f210589bda404913a7f456f39163a60d P 2ebc7147b38d42c183f6c7b6a62db075: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9432d0f6c60491b8a20b8dcf3561c84" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39607 } } peers { permanent_uuid: "410592fd61c042c9a956aada0e560a12" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45795 } } peers { permanent_uuid: "2ebc7147b38d42c183f6c7b6a62db075" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46789 } } }
I20260619 16:44:04.720048 8475 tablet_replica.cc:442] T f210589bda404913a7f456f39163a60d P 2ebc7147b38d42c183f6c7b6a62db075: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 16:44:04.719547 8468 ts_tablet_manager.cc:1428] T f210589bda404913a7f456f39163a60d P f9432d0f6c60491b8a20b8dcf3561c84: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260619 16:44:04.852828 8475 raft_consensus.cc:482] T f210589bda404913a7f456f39163a60d P 2ebc7147b38d42c183f6c7b6a62db075 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 16:44:04.853036 8475 raft_consensus.cc:504] T f210589bda404913a7f456f39163a60d P 2ebc7147b38d42c183f6c7b6a62db075 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9432d0f6c60491b8a20b8dcf3561c84" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39607 } } peers { permanent_uuid: "410592fd61c042c9a956aada0e560a12" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45795 } } peers { permanent_uuid: "2ebc7147b38d42c183f6c7b6a62db075" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46789 } }
I20260619 16:44:04.854470 8475 leader_election.cc:288] T f210589bda404913a7f456f39163a60d P 2ebc7147b38d42c183f6c7b6a62db075 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f9432d0f6c60491b8a20b8dcf3561c84 (127.0.97.1:39607), 410592fd61c042c9a956aada0e560a12 (127.0.97.2:45795)
I20260619 16:44:04.860255 8510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:44:04.854839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:40415 (local address 127.0.97.2:45795)
0619 16:44:04.855750 (+ 911us) server_negotiation.cc:206] Beginning negotiation
0619 16:44:04.855755 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:44:04.855769 (+ 14us) server_negotiation.cc:407] Connection header received
0619 16:44:04.855823 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:44:04.855826 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:44:04.855879 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:44:04.855965 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:44:04.856373 (+ 408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:44:04.857066 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:44:04.858627 (+ 1561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:44:04.859331 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:44:04.859441 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:44:04.859534 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0619 16:44:04.859802 (+ 268us) server_negotiation.cc:299] Negotiation successful
0619 16:44:04.859940 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":826,"thread_start_us":733,"threads_started":1}
I20260619 16:44:04.860270 8509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:44:04.854685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:45795 (local address 127.0.97.3:40415)
0619 16:44:04.855186 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0619 16:44:04.855207 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 16:44:04.855367 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:44:04.856002 (+ 635us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:44:04.856007 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:44:04.856032 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:44:04.856216 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:44:04.856224 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:44:04.857185 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:44:04.857191 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:44:04.858484 (+ 1293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:44:04.858496 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:44:04.859470 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:44:04.859475 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:44:04.859612 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:44:04.859666 (+ 54us) client_negotiation.cc:769] Sending connection context
0619 16:44:04.859760 (+ 94us) client_negotiation.cc:240] Negotiation successful
0619 16:44:04.859894 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":371,"mutex_wait_us":2,"thread_start_us":233,"threads_started":1}
I20260619 16:44:04.860407 8512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:44:04.854685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39607 (local address 127.0.97.3:47127)
0619 16:44:04.855758 (+ 1073us) negotiation.cc:107] Waiting for socket to connect
0619 16:44:04.855772 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 16:44:04.855908 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:44:04.856196 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:44:04.856200 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:44:04.856223 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:44:04.856465 (+ 242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:44:04.856473 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:44:04.857359 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:44:04.857362 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:44:04.858645 (+ 1283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:44:04.858656 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:44:04.859669 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:44:04.859675 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:44:04.859843 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:44:04.859875 (+ 32us) client_negotiation.cc:769] Sending connection context
0619 16:44:04.859952 (+ 77us) client_negotiation.cc:240] Negotiation successful
0619 16:44:04.860072 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":972,"mutex_wait_us":309,"thread_start_us":529,"threads_started":1}
I20260619 16:44:04.860972 8511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:44:04.854842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:47127 (local address 127.0.97.1:39607)
0619 16:44:04.855410 (+ 568us) server_negotiation.cc:206] Beginning negotiation
0619 16:44:04.855417 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:44:04.855850 (+ 433us) server_negotiation.cc:407] Connection header received
0619 16:44:04.856022 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:44:04.856026 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:44:04.856087 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:44:04.856163 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:44:04.856593 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:44:04.857227 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:44:04.858788 (+ 1561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:44:04.859512 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:44:04.859619 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:44:04.859710 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0619 16:44:04.860628 (+ 918us) server_negotiation.cc:299] Negotiation successful
0619 16:44:04.860764 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":452,"thread_start_us":359,"threads_started":1}
I20260619 16:44:04.860982 8254 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f210589bda404913a7f456f39163a60d" candidate_uuid: "2ebc7147b38d42c183f6c7b6a62db075" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "410592fd61c042c9a956aada0e560a12" is_pre_election: true
I20260619 16:44:04.861218 8254 raft_consensus.cc:2454] T f210589bda404913a7f456f39163a60d P 410592fd61c042c9a956aada0e560a12 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2ebc7147b38d42c183f6c7b6a62db075 in term 0.
I20260619 16:44:04.861645 8327 leader_election.cc:302] T f210589bda404913a7f456f39163a60d P 2ebc7147b38d42c183f6c7b6a62db075 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2ebc7147b38d42c183f6c7b6a62db075, 410592fd61c042c9a956aada0e560a12; no voters:
I20260619 16:44:04.861639 8121 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f210589bda404913a7f456f39163a60d" candidate_uuid: "2ebc7147b38d42c183f6c7b6a62db075" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f9432d0f6c60491b8a20b8dcf3561c84" is_pre_election: true
I20260619 16:44:04.862002 8475 raft_consensus.cc:2781] T f210589bda404913a7f456f39163a60d P 2ebc7147b38d42c183f6c7b6a62db075 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 16:44:04.861974 8121 raft_consensus.cc:2454] T f210589bda404913a7f456f39163a60d P f9432d0f6c60491b8a20b8dcf3561c84 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2ebc7147b38d42c183f6c7b6a62db075 in term 0.
I20260619 16:44:04.862110 8475 raft_consensus.cc:482] T f210589bda404913a7f456f39163a60d P 2ebc7147b38d42c183f6c7b6a62db075 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 16:44:04.862207 8475 raft_consensus.cc:3037] T f210589bda404913a7f456f39163a60d P 2ebc7147b38d42c183f6c7b6a62db075 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:44:04.864431 8475 raft_consensus.cc:504] T f210589bda404913a7f456f39163a60d P 2ebc7147b38d42c183f6c7b6a62db075 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9432d0f6c60491b8a20b8dcf3561c84" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39607 } } peers { permanent_uuid: "410592fd61c042c9a956aada0e560a12" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45795 } } peers { permanent_uuid: "2ebc7147b38d42c183f6c7b6a62db075" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46789 } }
I20260619 16:44:04.864946 8475 leader_election.cc:288] T f210589bda404913a7f456f39163a60d P 2ebc7147b38d42c183f6c7b6a62db075 [CANDIDATE]: Term 1 election: Requested vote from peers f9432d0f6c60491b8a20b8dcf3561c84 (127.0.97.1:39607), 410592fd61c042c9a956aada0e560a12 (127.0.97.2:45795)
I20260619 16:44:04.865293 8121 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f210589bda404913a7f456f39163a60d" candidate_uuid: "2ebc7147b38d42c183f6c7b6a62db075" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f9432d0f6c60491b8a20b8dcf3561c84"
I20260619 16:44:04.865370 8254 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f210589bda404913a7f456f39163a60d" candidate_uuid: "2ebc7147b38d42c183f6c7b6a62db075" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "410592fd61c042c9a956aada0e560a12"
I20260619 16:44:04.865437 8121 raft_consensus.cc:3037] T f210589bda404913a7f456f39163a60d P f9432d0f6c60491b8a20b8dcf3561c84 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:44:04.865567 8254 raft_consensus.cc:3037] T f210589bda404913a7f456f39163a60d P 410592fd61c042c9a956aada0e560a12 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:44:04.867298 8121 raft_consensus.cc:2454] T f210589bda404913a7f456f39163a60d P f9432d0f6c60491b8a20b8dcf3561c84 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2ebc7147b38d42c183f6c7b6a62db075 in term 1.
I20260619 16:44:04.867419 8254 raft_consensus.cc:2454] T f210589bda404913a7f456f39163a60d P 410592fd61c042c9a956aada0e560a12 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2ebc7147b38d42c183f6c7b6a62db075 in term 1.
I20260619 16:44:04.867624 8325 leader_election.cc:302] T f210589bda404913a7f456f39163a60d P 2ebc7147b38d42c183f6c7b6a62db075 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2ebc7147b38d42c183f6c7b6a62db075, f9432d0f6c60491b8a20b8dcf3561c84; no voters:
I20260619 16:44:04.867898 8475 raft_consensus.cc:2781] T f210589bda404913a7f456f39163a60d P 2ebc7147b38d42c183f6c7b6a62db075 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:44:04.868230 8475 raft_consensus.cc:686] T f210589bda404913a7f456f39163a60d P 2ebc7147b38d42c183f6c7b6a62db075 [term 1 LEADER]: Becoming Leader. State: Replica: 2ebc7147b38d42c183f6c7b6a62db075, State: Running, Role: LEADER
I20260619 16:44:04.868613 8475 consensus_queue.cc:237] T f210589bda404913a7f456f39163a60d P 2ebc7147b38d42c183f6c7b6a62db075 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9432d0f6c60491b8a20b8dcf3561c84" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39607 } } peers { permanent_uuid: "410592fd61c042c9a956aada0e560a12" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45795 } } peers { permanent_uuid: "2ebc7147b38d42c183f6c7b6a62db075" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46789 } }
I20260619 16:44:04.870205 8513 tablet_replica.cc:439] T f210589bda404913a7f456f39163a60d P 2ebc7147b38d42c183f6c7b6a62db075: TxnStatusTablet state changed. Reason: New leader 2ebc7147b38d42c183f6c7b6a62db075. Latest consensus state: current_term: 1 leader_uuid: "2ebc7147b38d42c183f6c7b6a62db075" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9432d0f6c60491b8a20b8dcf3561c84" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39607 } } peers { permanent_uuid: "410592fd61c042c9a956aada0e560a12" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45795 } } peers { permanent_uuid: "2ebc7147b38d42c183f6c7b6a62db075" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46789 } } }
I20260619 16:44:04.870472 8513 tablet_replica.cc:442] T f210589bda404913a7f456f39163a60d P 2ebc7147b38d42c183f6c7b6a62db075: This TxnStatusTablet replica's current role is: LEADER
I20260619 16:44:04.871135 8515 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260619 16:44:04.871351 8515 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260619 16:44:04.871909 7959 catalog_manager.cc:5495] T f210589bda404913a7f456f39163a60d P 2ebc7147b38d42c183f6c7b6a62db075 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2ebc7147b38d42c183f6c7b6a62db075 (127.0.97.3). New cstate: current_term: 1 leader_uuid: "2ebc7147b38d42c183f6c7b6a62db075" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9432d0f6c60491b8a20b8dcf3561c84" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39607 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "410592fd61c042c9a956aada0e560a12" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45795 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2ebc7147b38d42c183f6c7b6a62db075" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46789 } health_report { overall_health: HEALTHY } } }
I20260619 16:44:04.910904 8518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:44:04.905364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60336 (local address 127.0.97.3:46789)
0619 16:44:04.905997 (+ 633us) server_negotiation.cc:206] Beginning negotiation
0619 16:44:04.906001 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:44:04.906020 (+ 19us) server_negotiation.cc:407] Connection header received
0619 16:44:04.906095 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:44:04.906098 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:44:04.906172 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:44:04.906309 (+ 137us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 16:44:04.906680 (+ 371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:44:04.907496 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:44:04.908721 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:44:04.909158 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:44:04.909273 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:44:04.909694 (+ 421us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 16:44:04.910303 (+ 609us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 16:44:04.910449 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0619 16:44:04.910584 (+ 135us) server_negotiation.cc:299] Negotiation successful
0619 16:44:04.910651 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":526,"thread_start_us":398,"threads_started":1}
I20260619 16:44:04.910995 8505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:44:04.905266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:46789 (local address 127.0.0.1:60336)
0619 16:44:04.905459 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0619 16:44:04.905476 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 16:44:04.905580 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:44:04.906345 (+ 765us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:44:04.906349 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:44:04.906362 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 16:44:04.906551 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:44:04.906561 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:44:04.907619 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:44:04.907625 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:44:04.908596 (+ 971us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:44:04.908607 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:44:04.909292 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:44:04.909298 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:44:04.909449 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:44:04.909499 (+ 50us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 16:44:04.910422 (+ 923us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 16:44:04.910444 (+ 22us) client_negotiation.cc:769] Sending connection context
0619 16:44:04.910543 (+ 99us) client_negotiation.cc:240] Negotiation successful
0619 16:44:04.910689 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40}
I20260619 16:44:04.915102 8254 raft_consensus.cc:1261] T f210589bda404913a7f456f39163a60d P 410592fd61c042c9a956aada0e560a12 [term 1 FOLLOWER]: Refusing update from remote peer 2ebc7147b38d42c183f6c7b6a62db075: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:44:04.915555 8121 raft_consensus.cc:1261] T f210589bda404913a7f456f39163a60d P f9432d0f6c60491b8a20b8dcf3561c84 [term 1 FOLLOWER]: Refusing update from remote peer 2ebc7147b38d42c183f6c7b6a62db075: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:44:04.915771 8513 consensus_queue.cc:1035] T f210589bda404913a7f456f39163a60d P 2ebc7147b38d42c183f6c7b6a62db075 [LEADER]: Connected to new peer: Peer: permanent_uuid: "410592fd61c042c9a956aada0e560a12" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:44:04.916189 8475 consensus_queue.cc:1035] T f210589bda404913a7f456f39163a60d P 2ebc7147b38d42c183f6c7b6a62db075 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f9432d0f6c60491b8a20b8dcf3561c84" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:44:04.920208 8476 tablet_replica.cc:439] T f210589bda404913a7f456f39163a60d P 410592fd61c042c9a956aada0e560a12: TxnStatusTablet state changed. Reason: New leader 2ebc7147b38d42c183f6c7b6a62db075. Latest consensus state: current_term: 1 leader_uuid: "2ebc7147b38d42c183f6c7b6a62db075" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9432d0f6c60491b8a20b8dcf3561c84" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39607 } } peers { permanent_uuid: "410592fd61c042c9a956aada0e560a12" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45795 } } peers { permanent_uuid: "2ebc7147b38d42c183f6c7b6a62db075" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46789 } } }
I20260619 16:44:04.920496 8476 tablet_replica.cc:442] T f210589bda404913a7f456f39163a60d P 410592fd61c042c9a956aada0e560a12: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 16:44:04.921351 8474 tablet_replica.cc:439] T f210589bda404913a7f456f39163a60d P f9432d0f6c60491b8a20b8dcf3561c84: TxnStatusTablet state changed. Reason: New leader 2ebc7147b38d42c183f6c7b6a62db075. Latest consensus state: current_term: 1 leader_uuid: "2ebc7147b38d42c183f6c7b6a62db075" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9432d0f6c60491b8a20b8dcf3561c84" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39607 } } peers { permanent_uuid: "410592fd61c042c9a956aada0e560a12" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45795 } } peers { permanent_uuid: "2ebc7147b38d42c183f6c7b6a62db075" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46789 } } }
I20260619 16:44:04.921650 8474 tablet_replica.cc:442] T f210589bda404913a7f456f39163a60d P f9432d0f6c60491b8a20b8dcf3561c84: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 16:44:04.924644 8522 mvcc.cc:205] Tried to move back new op lower bound from 7298610974365675520 to 7298610974184742912. Current Snapshot: MvccSnapshot[applied={T|T < 7298610974365675520}]
I20260619 16:44:04.925697 8475 tablet_replica.cc:439] T f210589bda404913a7f456f39163a60d P 2ebc7147b38d42c183f6c7b6a62db075: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2ebc7147b38d42c183f6c7b6a62db075" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9432d0f6c60491b8a20b8dcf3561c84" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39607 } } peers { permanent_uuid: "410592fd61c042c9a956aada0e560a12" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45795 } } peers { permanent_uuid: "2ebc7147b38d42c183f6c7b6a62db075" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46789 } } }
I20260619 16:44:04.925987 8475 tablet_replica.cc:442] T f210589bda404913a7f456f39163a60d P 2ebc7147b38d42c183f6c7b6a62db075: This TxnStatusTablet replica's current role is: LEADER
I20260619 16:44:04.926041 8476 tablet_replica.cc:439] T f210589bda404913a7f456f39163a60d P 410592fd61c042c9a956aada0e560a12: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2ebc7147b38d42c183f6c7b6a62db075" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9432d0f6c60491b8a20b8dcf3561c84" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39607 } } peers { permanent_uuid: "410592fd61c042c9a956aada0e560a12" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45795 } } peers { permanent_uuid: "2ebc7147b38d42c183f6c7b6a62db075" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46789 } } }
I20260619 16:44:04.926257 8476 tablet_replica.cc:442] T f210589bda404913a7f456f39163a60d P 410592fd61c042c9a956aada0e560a12: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 16:44:04.926237 8475 tablet_replica.cc:439] T f210589bda404913a7f456f39163a60d P 2ebc7147b38d42c183f6c7b6a62db075: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2ebc7147b38d42c183f6c7b6a62db075" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9432d0f6c60491b8a20b8dcf3561c84" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39607 } } peers { permanent_uuid: "410592fd61c042c9a956aada0e560a12" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45795 } } peers { permanent_uuid: "2ebc7147b38d42c183f6c7b6a62db075" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46789 } } }
I20260619 16:44:04.926435 8475 tablet_replica.cc:442] T f210589bda404913a7f456f39163a60d P 2ebc7147b38d42c183f6c7b6a62db075: This TxnStatusTablet replica's current role is: LEADER
I20260619 16:44:04.930269 8474 tablet_replica.cc:439] T f210589bda404913a7f456f39163a60d P f9432d0f6c60491b8a20b8dcf3561c84: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2ebc7147b38d42c183f6c7b6a62db075" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9432d0f6c60491b8a20b8dcf3561c84" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39607 } } peers { permanent_uuid: "410592fd61c042c9a956aada0e560a12" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45795 } } peers { permanent_uuid: "2ebc7147b38d42c183f6c7b6a62db075" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46789 } } }
I20260619 16:44:04.930536 8474 tablet_replica.cc:442] T f210589bda404913a7f456f39163a60d P f9432d0f6c60491b8a20b8dcf3561c84: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 16:44:04.945291 8518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:44:04.939009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:55773 (local address 127.0.97.3:46789)
0619 16:44:04.939179 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0619 16:44:04.939185 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:44:04.939693 (+ 508us) server_negotiation.cc:407] Connection header received
0619 16:44:04.939885 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:44:04.939889 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:44:04.939965 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:44:04.940036 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 16:44:04.940582 (+ 546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:44:04.941317 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:44:04.943267 (+ 1950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:44:04.943962 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:44:04.944126 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:44:04.944689 (+ 563us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 16:44:04.944798 (+ 109us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 16:44:04.944872 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0619 16:44:04.945003 (+ 131us) server_negotiation.cc:299] Negotiation successful
0619 16:44:04.945061 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260619 16:44:04.945477 8529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:44:04.938915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:46789 (local address 127.0.97.1:55773)
0619 16:44:04.939597 (+ 682us) negotiation.cc:107] Waiting for socket to connect
0619 16:44:04.939617 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 16:44:04.939759 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:44:04.940063 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:44:04.940068 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:44:04.940081 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 16:44:04.940416 (+ 335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:44:04.940428 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:44:04.941432 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:44:04.941438 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:44:04.943122 (+ 1684us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:44:04.943138 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:44:04.944188 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:44:04.944196 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:44:04.944458 (+ 262us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:44:04.944528 (+ 70us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 16:44:04.944889 (+ 361us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 16:44:04.944899 (+ 10us) client_negotiation.cc:769] Sending connection context
0619 16:44:04.944971 (+ 72us) client_negotiation.cc:240] Negotiation successful
0619 16:44:04.945110 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":527,"thread_start_us":408,"threads_started":1}
I20260619 16:44:04.955935 8394 raft_consensus.cc:1261] T 20e0d115a7d04b30b0107038155568ee P 2ebc7147b38d42c183f6c7b6a62db075 [term 1 FOLLOWER]: Refusing update from remote peer f9432d0f6c60491b8a20b8dcf3561c84: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:44:04.956427 8254 raft_consensus.cc:1261] T 20e0d115a7d04b30b0107038155568ee P 410592fd61c042c9a956aada0e560a12 [term 1 FOLLOWER]: Refusing update from remote peer f9432d0f6c60491b8a20b8dcf3561c84: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:44:04.956436 8474 consensus_queue.cc:1035] T 20e0d115a7d04b30b0107038155568ee P f9432d0f6c60491b8a20b8dcf3561c84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ebc7147b38d42c183f6c7b6a62db075" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:44:04.956861 8479 consensus_queue.cc:1035] T 20e0d115a7d04b30b0107038155568ee P f9432d0f6c60491b8a20b8dcf3561c84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "410592fd61c042c9a956aada0e560a12" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:44:04.963254 8520 mvcc.cc:205] Tried to move back new op lower bound from 7298610974533173248 to 7298610973044023296. Current Snapshot: MvccSnapshot[applied={T|T < 7298610974533173248}]
I20260619 16:44:04.999469 8511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:44:04.987221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:39285 (local address 127.0.97.1:39607)
0619 16:44:04.987400 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0619 16:44:04.987406 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:44:04.987747 (+ 341us) server_negotiation.cc:407] Connection header received
0619 16:44:04.988002 (+ 255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:44:04.988007 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:44:04.988082 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:44:04.988161 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 16:44:04.988892 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:44:04.989928 (+ 1036us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:44:04.991825 (+ 1897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:44:04.992455 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:44:04.992563 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:44:04.998725 (+ 6162us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 16:44:04.998855 (+ 130us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 16:44:04.998917 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0619 16:44:04.999136 (+ 219us) server_negotiation.cc:299] Negotiation successful
0619 16:44:04.999221 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260619 16:44:04.999658 8536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:44:04.986840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39607 (local address 127.0.97.3:39285)
0619 16:44:04.987654 (+ 814us) negotiation.cc:107] Waiting for socket to connect
0619 16:44:04.987670 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 16:44:04.987824 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:44:04.988314 (+ 490us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:44:04.988318 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:44:04.988330 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 16:44:04.988710 (+ 380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:44:04.988723 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:44:04.990330 (+ 1607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:44:04.990335 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:44:04.991679 (+ 1344us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:44:04.991691 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:44:04.998255 (+ 6564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:44:04.998261 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:44:04.998444 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:44:04.998515 (+ 71us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 16:44:04.999002 (+ 487us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 16:44:04.999012 (+ 10us) client_negotiation.cc:769] Sending connection context
0619 16:44:04.999090 (+ 78us) client_negotiation.cc:240] Negotiation successful
0619 16:44:04.999253 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":543,"thread_start_us":571,"threads_started":1}
I20260619 16:44:05.042011 7959 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48018:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:44:05.042255 7959 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48018:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:44:05.045048 7959 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ed5a208fb1504c2dbce0eceab08ad43c: Sending DeleteTablet for 3 replicas of tablet 20e0d115a7d04b30b0107038155568ee
I20260619 16:44:05.045713 8096 tablet_service.cc:1514] Processing DeleteTablet for tablet 20e0d115a7d04b30b0107038155568ee with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:44:05 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:40886
I20260619 16:44:05.046168 8235 tablet_service.cc:1514] Processing DeleteTablet for tablet 20e0d115a7d04b30b0107038155568ee with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:44:05 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:59360
I20260619 16:44:05.046730 8373 tablet_service.cc:1514] Processing DeleteTablet for tablet 20e0d115a7d04b30b0107038155568ee with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:44:05 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:60330
I20260619 16:44:05.056751 8539 tablet_replica.cc:330] T 20e0d115a7d04b30b0107038155568ee P 410592fd61c042c9a956aada0e560a12: stopping tablet replica
I20260619 16:44:05.057003 8539 raft_consensus.cc:2229] T 20e0d115a7d04b30b0107038155568ee P 410592fd61c042c9a956aada0e560a12 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 16:44:05.057328 8539 raft_consensus.cc:2258] T 20e0d115a7d04b30b0107038155568ee P 410592fd61c042c9a956aada0e560a12 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:44:05.058130 8538 tablet_replica.cc:330] T 20e0d115a7d04b30b0107038155568ee P f9432d0f6c60491b8a20b8dcf3561c84: stopping tablet replica
I20260619 16:44:05.058418 8539 ts_tablet_manager.cc:1905] T 20e0d115a7d04b30b0107038155568ee P 410592fd61c042c9a956aada0e560a12: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:44:05.058465 8538 raft_consensus.cc:2229] T 20e0d115a7d04b30b0107038155568ee P f9432d0f6c60491b8a20b8dcf3561c84 [term 1 LEADER]: Raft consensus shutting down.
I20260619 16:44:05.058938 8538 raft_consensus.cc:2258] T 20e0d115a7d04b30b0107038155568ee P f9432d0f6c60491b8a20b8dcf3561c84 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:44:05.060006 8538 ts_tablet_manager.cc:1905] T 20e0d115a7d04b30b0107038155568ee P f9432d0f6c60491b8a20b8dcf3561c84: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:44:05.061704 8540 tablet_replica.cc:330] T 20e0d115a7d04b30b0107038155568ee P 2ebc7147b38d42c183f6c7b6a62db075: stopping tablet replica
I20260619 16:44:05.063056 8540 raft_consensus.cc:2229] T 20e0d115a7d04b30b0107038155568ee P 2ebc7147b38d42c183f6c7b6a62db075 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 16:44:05.063494 8540 raft_consensus.cc:2258] T 20e0d115a7d04b30b0107038155568ee P 2ebc7147b38d42c183f6c7b6a62db075 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:44:05.064440 8540 ts_tablet_manager.cc:1905] T 20e0d115a7d04b30b0107038155568ee P 2ebc7147b38d42c183f6c7b6a62db075: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:44:05.066555 8538 ts_tablet_manager.cc:1918] T 20e0d115a7d04b30b0107038155568ee P f9432d0f6c60491b8a20b8dcf3561c84: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 16:44:05.066726 8538 log.cc:1196] T 20e0d115a7d04b30b0107038155568ee P f9432d0f6c60491b8a20b8dcf3561c84: Deleting WAL directory at /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887385903558-388-0/minicluster-data/ts-0/wal/wals/20e0d115a7d04b30b0107038155568ee
I20260619 16:44:05.067199 8538 ts_tablet_manager.cc:1939] T 20e0d115a7d04b30b0107038155568ee P f9432d0f6c60491b8a20b8dcf3561c84: Deleting consensus metadata
I20260619 16:44:05.068265 7943 catalog_manager.cc:4841] TS f9432d0f6c60491b8a20b8dcf3561c84 (127.0.97.1:39607): tablet 20e0d115a7d04b30b0107038155568ee (table test-table [id=d6515c8cd5554ba4ad1055ea387c6d93]) successfully deleted
I20260619 16:44:05.070703 8540 ts_tablet_manager.cc:1918] T 20e0d115a7d04b30b0107038155568ee P 2ebc7147b38d42c183f6c7b6a62db075: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 16:44:05.070706 8539 ts_tablet_manager.cc:1918] T 20e0d115a7d04b30b0107038155568ee P 410592fd61c042c9a956aada0e560a12: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 16:44:05.070819 8539 log.cc:1196] T 20e0d115a7d04b30b0107038155568ee P 410592fd61c042c9a956aada0e560a12: Deleting WAL directory at /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887385903558-388-0/minicluster-data/ts-1/wal/wals/20e0d115a7d04b30b0107038155568ee
I20260619 16:44:05.070942 8540 log.cc:1196] T 20e0d115a7d04b30b0107038155568ee P 2ebc7147b38d42c183f6c7b6a62db075: Deleting WAL directory at /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887385903558-388-0/minicluster-data/ts-2/wal/wals/20e0d115a7d04b30b0107038155568ee
I20260619 16:44:05.071257 8539 ts_tablet_manager.cc:1939] T 20e0d115a7d04b30b0107038155568ee P 410592fd61c042c9a956aada0e560a12: Deleting consensus metadata
I20260619 16:44:05.071377 8540 ts_tablet_manager.cc:1939] T 20e0d115a7d04b30b0107038155568ee P 2ebc7147b38d42c183f6c7b6a62db075: Deleting consensus metadata
I20260619 16:44:05.072306 7945 catalog_manager.cc:4841] TS 2ebc7147b38d42c183f6c7b6a62db075 (127.0.97.3:46789): tablet 20e0d115a7d04b30b0107038155568ee (table test-table [id=d6515c8cd5554ba4ad1055ea387c6d93]) successfully deleted
I20260619 16:44:05.072516 7945 catalog_manager.cc:4841] TS 410592fd61c042c9a956aada0e560a12 (127.0.97.2:45795): tablet 20e0d115a7d04b30b0107038155568ee (table test-table [id=d6515c8cd5554ba4ad1055ea387c6d93]) successfully deleted
Jun 19 16:44:05 dist-test-slave-wvz7 krb5kdc[7914](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887445, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 19 16:44:05 dist-test-slave-wvz7 krb5kdc[7914](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:44:05 dist-test-slave-wvz7 krb5kdc[7914](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
W20260619 16:44:05.118825 8504 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:44:05.108564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48036 (local address 127.0.97.62:39907)
0619 16:44:05.108765 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0619 16:44:05.108772 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:44:05.109089 (+ 317us) server_negotiation.cc:407] Connection header received
0619 16:44:05.109234 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:44:05.109240 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:44:05.109321 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:44:05.109436 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:44:05.110704 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:44:05.111779 (+ 1075us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:44:05.113007 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:44:05.113616 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:44:05.113753 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:44:05.118565 (+ 4812us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48036: BlockingRecv error: recv got EOF from 127.0.0.1:48036 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Jun 19 16:44:05 dist-test-slave-wvz7 krb5kdc[7914](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887445, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260619 16:44:05.133839 8504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:44:05.121370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48044 (local address 127.0.97.62:39907)
0619 16:44:05.121527 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0619 16:44:05.121532 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:44:05.121802 (+ 270us) server_negotiation.cc:407] Connection header received
0619 16:44:05.121996 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:44:05.122002 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:44:05.122063 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:44:05.122149 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:44:05.123237 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:44:05.123927 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:44:05.125081 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:44:05.125498 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:44:05.125599 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:44:05.129125 (+ 3526us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:44:05.129129 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:44:05.129132 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:44:05.129163 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:44:05.131573 (+ 2410us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:44:05.132127 (+ 554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:44:05.132132 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:44:05.132134 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:44:05.132268 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:44:05.132582 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:44:05.132587 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:44:05.132590 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:44:05.132835 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:44:05.133057 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0619 16:44:05.133309 (+ 252us) server_negotiation.cc:299] Negotiation successful
0619 16:44:05.133518 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260619 16:44:05.136941 7959 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48044:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 16:44:05.137517 7959 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 16:44:05.143558 8096 tablet_service.cc:1467] Processing CreateTablet for tablet d6b248113030427186ea796758fbbee1 (DEFAULT_TABLE table=test-table [id=36dcae5d9e3a490abbcd32c42ee5dae5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:44:05.143985 8096 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d6b248113030427186ea796758fbbee1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:44:05.143918 8373 tablet_service.cc:1467] Processing CreateTablet for tablet d6b248113030427186ea796758fbbee1 (DEFAULT_TABLE table=test-table [id=36dcae5d9e3a490abbcd32c42ee5dae5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:44:05.144318 8373 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d6b248113030427186ea796758fbbee1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:44:05.146106 8235 tablet_service.cc:1467] Processing CreateTablet for tablet d6b248113030427186ea796758fbbee1 (DEFAULT_TABLE table=test-table [id=36dcae5d9e3a490abbcd32c42ee5dae5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:44:05.146497 8235 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d6b248113030427186ea796758fbbee1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:44:05.148838 8470 tablet_bootstrap.cc:492] T d6b248113030427186ea796758fbbee1 P 2ebc7147b38d42c183f6c7b6a62db075: Bootstrap starting.
I20260619 16:44:05.150461 8470 tablet_bootstrap.cc:654] T d6b248113030427186ea796758fbbee1 P 2ebc7147b38d42c183f6c7b6a62db075: Neither blocks nor log segments found. Creating new log.
I20260619 16:44:05.152412 8471 tablet_bootstrap.cc:492] T d6b248113030427186ea796758fbbee1 P 410592fd61c042c9a956aada0e560a12: Bootstrap starting.
I20260619 16:44:05.153712 8468 tablet_bootstrap.cc:492] T d6b248113030427186ea796758fbbee1 P f9432d0f6c60491b8a20b8dcf3561c84: Bootstrap starting.
I20260619 16:44:05.153874 8471 tablet_bootstrap.cc:654] T d6b248113030427186ea796758fbbee1 P 410592fd61c042c9a956aada0e560a12: Neither blocks nor log segments found. Creating new log.
I20260619 16:44:05.155171 8468 tablet_bootstrap.cc:654] T d6b248113030427186ea796758fbbee1 P f9432d0f6c60491b8a20b8dcf3561c84: Neither blocks nor log segments found. Creating new log.
I20260619 16:44:05.155936 8470 tablet_bootstrap.cc:492] T d6b248113030427186ea796758fbbee1 P 2ebc7147b38d42c183f6c7b6a62db075: No bootstrap required, opened a new log
I20260619 16:44:05.156088 8470 ts_tablet_manager.cc:1397] T d6b248113030427186ea796758fbbee1 P 2ebc7147b38d42c183f6c7b6a62db075: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.003s
I20260619 16:44:05.157078 8470 raft_consensus.cc:348] T d6b248113030427186ea796758fbbee1 P 2ebc7147b38d42c183f6c7b6a62db075 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9432d0f6c60491b8a20b8dcf3561c84" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39607 } } peers { permanent_uuid: "410592fd61c042c9a956aada0e560a12" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45795 } } peers { permanent_uuid: "2ebc7147b38d42c183f6c7b6a62db075" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46789 } }
I20260619 16:44:05.157358 8470 raft_consensus.cc:374] T d6b248113030427186ea796758fbbee1 P 2ebc7147b38d42c183f6c7b6a62db075 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:44:05.157460 8470 raft_consensus.cc:729] T d6b248113030427186ea796758fbbee1 P 2ebc7147b38d42c183f6c7b6a62db075 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ebc7147b38d42c183f6c7b6a62db075, State: Initialized, Role: FOLLOWER
I20260619 16:44:05.158403 8470 consensus_queue.cc:260] T d6b248113030427186ea796758fbbee1 P 2ebc7147b38d42c183f6c7b6a62db075 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9432d0f6c60491b8a20b8dcf3561c84" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39607 } } peers { permanent_uuid: "410592fd61c042c9a956aada0e560a12" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45795 } } peers { permanent_uuid: "2ebc7147b38d42c183f6c7b6a62db075" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46789 } }
I20260619 16:44:05.159812 8470 ts_tablet_manager.cc:1428] T d6b248113030427186ea796758fbbee1 P 2ebc7147b38d42c183f6c7b6a62db075: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20260619 16:44:05.161114 8471 tablet_bootstrap.cc:492] T d6b248113030427186ea796758fbbee1 P 410592fd61c042c9a956aada0e560a12: No bootstrap required, opened a new log
I20260619 16:44:05.161302 8471 ts_tablet_manager.cc:1397] T d6b248113030427186ea796758fbbee1 P 410592fd61c042c9a956aada0e560a12: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.007s
I20260619 16:44:05.162549 8471 raft_consensus.cc:348] T d6b248113030427186ea796758fbbee1 P 410592fd61c042c9a956aada0e560a12 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9432d0f6c60491b8a20b8dcf3561c84" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39607 } } peers { permanent_uuid: "410592fd61c042c9a956aada0e560a12" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45795 } } peers { permanent_uuid: "2ebc7147b38d42c183f6c7b6a62db075" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46789 } }
I20260619 16:44:05.162823 8471 raft_consensus.cc:374] T d6b248113030427186ea796758fbbee1 P 410592fd61c042c9a956aada0e560a12 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:44:05.162948 8471 raft_consensus.cc:729] T d6b248113030427186ea796758fbbee1 P 410592fd61c042c9a956aada0e560a12 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 410592fd61c042c9a956aada0e560a12, State: Initialized, Role: FOLLOWER
I20260619 16:44:05.163175 8471 consensus_queue.cc:260] T d6b248113030427186ea796758fbbee1 P 410592fd61c042c9a956aada0e560a12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9432d0f6c60491b8a20b8dcf3561c84" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39607 } } peers { permanent_uuid: "410592fd61c042c9a956aada0e560a12" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45795 } } peers { permanent_uuid: "2ebc7147b38d42c183f6c7b6a62db075" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46789 } }
I20260619 16:44:05.163852 8471 ts_tablet_manager.cc:1428] T d6b248113030427186ea796758fbbee1 P 410592fd61c042c9a956aada0e560a12: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20260619 16:44:05.165347 8468 tablet_bootstrap.cc:492] T d6b248113030427186ea796758fbbee1 P f9432d0f6c60491b8a20b8dcf3561c84: No bootstrap required, opened a new log
I20260619 16:44:05.165491 8468 ts_tablet_manager.cc:1397] T d6b248113030427186ea796758fbbee1 P f9432d0f6c60491b8a20b8dcf3561c84: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.007s
I20260619 16:44:05.166244 8468 raft_consensus.cc:348] T d6b248113030427186ea796758fbbee1 P f9432d0f6c60491b8a20b8dcf3561c84 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9432d0f6c60491b8a20b8dcf3561c84" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39607 } } peers { permanent_uuid: "410592fd61c042c9a956aada0e560a12" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45795 } } peers { permanent_uuid: "2ebc7147b38d42c183f6c7b6a62db075" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46789 } }
I20260619 16:44:05.166491 8468 raft_consensus.cc:374] T d6b248113030427186ea796758fbbee1 P f9432d0f6c60491b8a20b8dcf3561c84 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:44:05.166579 8468 raft_consensus.cc:729] T d6b248113030427186ea796758fbbee1 P f9432d0f6c60491b8a20b8dcf3561c84 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f9432d0f6c60491b8a20b8dcf3561c84, State: Initialized, Role: FOLLOWER
I20260619 16:44:05.166854 8468 consensus_queue.cc:260] T d6b248113030427186ea796758fbbee1 P f9432d0f6c60491b8a20b8dcf3561c84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9432d0f6c60491b8a20b8dcf3561c84" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39607 } } peers { permanent_uuid: "410592fd61c042c9a956aada0e560a12" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45795 } } peers { permanent_uuid: "2ebc7147b38d42c183f6c7b6a62db075" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46789 } }
I20260619 16:44:05.167481 8468 ts_tablet_manager.cc:1428] T d6b248113030427186ea796758fbbee1 P f9432d0f6c60491b8a20b8dcf3561c84: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260619 16:44:05.467504 8476 raft_consensus.cc:482] T d6b248113030427186ea796758fbbee1 P 410592fd61c042c9a956aada0e560a12 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 16:44:05.467691 8476 raft_consensus.cc:504] T d6b248113030427186ea796758fbbee1 P 410592fd61c042c9a956aada0e560a12 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9432d0f6c60491b8a20b8dcf3561c84" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39607 } } peers { permanent_uuid: "410592fd61c042c9a956aada0e560a12" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45795 } } peers { permanent_uuid: "2ebc7147b38d42c183f6c7b6a62db075" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46789 } }
I20260619 16:44:05.469111 8476 leader_election.cc:288] T d6b248113030427186ea796758fbbee1 P 410592fd61c042c9a956aada0e560a12 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f9432d0f6c60491b8a20b8dcf3561c84 (127.0.97.1:39607), 2ebc7147b38d42c183f6c7b6a62db075 (127.0.97.3:46789)
I20260619 16:44:05.477240 8511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:44:05.469453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:51265 (local address 127.0.97.1:39607)
0619 16:44:05.469634 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0619 16:44:05.469641 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:44:05.470471 (+ 830us) server_negotiation.cc:407] Connection header received
0619 16:44:05.470608 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:44:05.470618 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:44:05.470696 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:44:05.470783 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:44:05.471446 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:44:05.472432 (+ 986us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:44:05.475107 (+ 2675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:44:05.476230 (+ 1123us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:44:05.476374 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:44:05.476455 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0619 16:44:05.476880 (+ 425us) server_negotiation.cc:299] Negotiation successful
0619 16:44:05.476984 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260619 16:44:05.477376 8558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:44:05.469596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:42105 (local address 127.0.97.3:46789)
0619 16:44:05.470509 (+ 913us) server_negotiation.cc:206] Beginning negotiation
0619 16:44:05.470515 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:44:05.470531 (+ 16us) server_negotiation.cc:407] Connection header received
0619 16:44:05.470601 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:44:05.470606 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:44:05.470677 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:44:05.470773 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:44:05.471373 (+ 600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:44:05.472433 (+ 1060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:44:05.475108 (+ 2675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:44:05.476244 (+ 1136us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:44:05.476351 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:44:05.476450 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0619 16:44:05.477014 (+ 564us) server_negotiation.cc:299] Negotiation successful
0619 16:44:05.477144 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":785,"thread_start_us":690,"threads_started":1}
I20260619 16:44:05.477499 8557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:44:05.469394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39607 (local address 127.0.97.2:51265)
0619 16:44:05.470358 (+ 964us) negotiation.cc:107] Waiting for socket to connect
0619 16:44:05.470374 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 16:44:05.470481 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:44:05.470821 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:44:05.470825 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:44:05.470864 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:44:05.471298 (+ 434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:44:05.471308 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:44:05.472569 (+ 1261us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:44:05.472578 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:44:05.474961 (+ 2383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:44:05.474975 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:44:05.476466 (+ 1491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:44:05.476473 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:44:05.476651 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:44:05.476738 (+ 87us) client_negotiation.cc:769] Sending connection context
0619 16:44:05.476850 (+ 112us) client_negotiation.cc:240] Negotiation successful
0619 16:44:05.476980 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":816,"mutex_wait_us":170,"spinlock_wait_cycles":4608,"thread_start_us":657,"threads_started":1}
I20260619 16:44:05.477811 8556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:44:05.469373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:46789 (local address 127.0.97.2:42105)
0619 16:44:05.470044 (+ 671us) negotiation.cc:107] Waiting for socket to connect
0619 16:44:05.470065 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 16:44:05.470201 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:44:05.470816 (+ 615us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:44:05.470820 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:44:05.470856 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:44:05.471166 (+ 310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:44:05.471181 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:44:05.472582 (+ 1401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:44:05.472586 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:44:05.474961 (+ 2375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:44:05.474975 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:44:05.476655 (+ 1680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:44:05.476666 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:44:05.476854 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:44:05.476887 (+ 33us) client_negotiation.cc:769] Sending connection context
0619 16:44:05.476971 (+ 84us) client_negotiation.cc:240] Negotiation successful
0619 16:44:05.477083 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":558,"mutex_wait_us":94,"spinlock_wait_cycles":1408,"thread_start_us":412,"threads_started":1}
I20260619 16:44:05.478248 8121 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6b248113030427186ea796758fbbee1" candidate_uuid: "410592fd61c042c9a956aada0e560a12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f9432d0f6c60491b8a20b8dcf3561c84" is_pre_election: true
I20260619 16:44:05.478324 8394 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6b248113030427186ea796758fbbee1" candidate_uuid: "410592fd61c042c9a956aada0e560a12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ebc7147b38d42c183f6c7b6a62db075" is_pre_election: true
I20260619 16:44:05.478472 8121 raft_consensus.cc:2454] T d6b248113030427186ea796758fbbee1 P f9432d0f6c60491b8a20b8dcf3561c84 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 410592fd61c042c9a956aada0e560a12 in term 0.
I20260619 16:44:05.478550 8394 raft_consensus.cc:2454] T d6b248113030427186ea796758fbbee1 P 2ebc7147b38d42c183f6c7b6a62db075 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 410592fd61c042c9a956aada0e560a12 in term 0.
I20260619 16:44:05.479022 8188 leader_election.cc:302] T d6b248113030427186ea796758fbbee1 P 410592fd61c042c9a956aada0e560a12 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2ebc7147b38d42c183f6c7b6a62db075, 410592fd61c042c9a956aada0e560a12; no voters:
I20260619 16:44:05.479357 8476 raft_consensus.cc:2781] T d6b248113030427186ea796758fbbee1 P 410592fd61c042c9a956aada0e560a12 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 16:44:05.479477 8476 raft_consensus.cc:482] T d6b248113030427186ea796758fbbee1 P 410592fd61c042c9a956aada0e560a12 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 16:44:05.479595 8476 raft_consensus.cc:3037] T d6b248113030427186ea796758fbbee1 P 410592fd61c042c9a956aada0e560a12 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:44:05.482017 8476 raft_consensus.cc:504] T d6b248113030427186ea796758fbbee1 P 410592fd61c042c9a956aada0e560a12 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9432d0f6c60491b8a20b8dcf3561c84" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39607 } } peers { permanent_uuid: "410592fd61c042c9a956aada0e560a12" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45795 } } peers { permanent_uuid: "2ebc7147b38d42c183f6c7b6a62db075" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46789 } }
I20260619 16:44:05.482649 8476 leader_election.cc:288] T d6b248113030427186ea796758fbbee1 P 410592fd61c042c9a956aada0e560a12 [CANDIDATE]: Term 1 election: Requested vote from peers f9432d0f6c60491b8a20b8dcf3561c84 (127.0.97.1:39607), 2ebc7147b38d42c183f6c7b6a62db075 (127.0.97.3:46789)
I20260619 16:44:05.483052 8121 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6b248113030427186ea796758fbbee1" candidate_uuid: "410592fd61c042c9a956aada0e560a12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f9432d0f6c60491b8a20b8dcf3561c84"
I20260619 16:44:05.483117 8394 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6b248113030427186ea796758fbbee1" candidate_uuid: "410592fd61c042c9a956aada0e560a12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ebc7147b38d42c183f6c7b6a62db075"
I20260619 16:44:05.483289 8121 raft_consensus.cc:3037] T d6b248113030427186ea796758fbbee1 P f9432d0f6c60491b8a20b8dcf3561c84 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:44:05.483310 8394 raft_consensus.cc:3037] T d6b248113030427186ea796758fbbee1 P 2ebc7147b38d42c183f6c7b6a62db075 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:44:05.486176 8394 raft_consensus.cc:2454] T d6b248113030427186ea796758fbbee1 P 2ebc7147b38d42c183f6c7b6a62db075 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 410592fd61c042c9a956aada0e560a12 in term 1.
I20260619 16:44:05.486164 8121 raft_consensus.cc:2454] T d6b248113030427186ea796758fbbee1 P f9432d0f6c60491b8a20b8dcf3561c84 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 410592fd61c042c9a956aada0e560a12 in term 1.
I20260619 16:44:05.486594 8186 leader_election.cc:302] T d6b248113030427186ea796758fbbee1 P 410592fd61c042c9a956aada0e560a12 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 410592fd61c042c9a956aada0e560a12, f9432d0f6c60491b8a20b8dcf3561c84; no voters:
I20260619 16:44:05.486887 8476 raft_consensus.cc:2781] T d6b248113030427186ea796758fbbee1 P 410592fd61c042c9a956aada0e560a12 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:44:05.487305 8476 raft_consensus.cc:686] T d6b248113030427186ea796758fbbee1 P 410592fd61c042c9a956aada0e560a12 [term 1 LEADER]: Becoming Leader. State: Replica: 410592fd61c042c9a956aada0e560a12, State: Running, Role: LEADER
I20260619 16:44:05.487694 8476 consensus_queue.cc:237] T d6b248113030427186ea796758fbbee1 P 410592fd61c042c9a956aada0e560a12 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9432d0f6c60491b8a20b8dcf3561c84" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39607 } } peers { permanent_uuid: "410592fd61c042c9a956aada0e560a12" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45795 } } peers { permanent_uuid: "2ebc7147b38d42c183f6c7b6a62db075" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46789 } }
I20260619 16:44:05.491181 7959 catalog_manager.cc:5495] T d6b248113030427186ea796758fbbee1 P 410592fd61c042c9a956aada0e560a12 reported cstate change: term changed from 0 to 1, leader changed from <none> to 410592fd61c042c9a956aada0e560a12 (127.0.97.2). New cstate: current_term: 1 leader_uuid: "410592fd61c042c9a956aada0e560a12" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9432d0f6c60491b8a20b8dcf3561c84" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39607 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "410592fd61c042c9a956aada0e560a12" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45795 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2ebc7147b38d42c183f6c7b6a62db075" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46789 } health_report { overall_health: UNKNOWN } } }
I20260619 16:44:05.526929 8562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:44:05.521428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59376 (local address 127.0.97.2:45795)
0619 16:44:05.521945 (+ 517us) server_negotiation.cc:206] Beginning negotiation
0619 16:44:05.521949 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:44:05.521968 (+ 19us) server_negotiation.cc:407] Connection header received
0619 16:44:05.522049 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:44:05.522053 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:44:05.522126 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:44:05.522272 (+ 146us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 16:44:05.522643 (+ 371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:44:05.523443 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:44:05.524739 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:44:05.525146 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:44:05.525232 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:44:05.525671 (+ 439us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 16:44:05.526293 (+ 622us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 16:44:05.526434 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0619 16:44:05.526559 (+ 125us) server_negotiation.cc:299] Negotiation successful
0619 16:44:05.526620 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":400,"thread_start_us":247,"threads_started":1}
I20260619 16:44:05.530432 8121 raft_consensus.cc:1261] T d6b248113030427186ea796758fbbee1 P f9432d0f6c60491b8a20b8dcf3561c84 [term 1 FOLLOWER]: Refusing update from remote peer 410592fd61c042c9a956aada0e560a12: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:44:05.530439 8394 raft_consensus.cc:1261] T d6b248113030427186ea796758fbbee1 P 2ebc7147b38d42c183f6c7b6a62db075 [term 1 FOLLOWER]: Refusing update from remote peer 410592fd61c042c9a956aada0e560a12: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:44:05.531114 8559 consensus_queue.cc:1035] T d6b248113030427186ea796758fbbee1 P 410592fd61c042c9a956aada0e560a12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f9432d0f6c60491b8a20b8dcf3561c84" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:44:05.531401 8476 consensus_queue.cc:1035] T d6b248113030427186ea796758fbbee1 P 410592fd61c042c9a956aada0e560a12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ebc7147b38d42c183f6c7b6a62db075" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:44:05.558599 7957 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48044:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:44:05.558842 7957 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48044:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:44:05.562379 7957 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ed5a208fb1504c2dbce0eceab08ad43c: Sending DeleteTablet for 3 replicas of tablet d6b248113030427186ea796758fbbee1
I20260619 16:44:05.563109 8096 tablet_service.cc:1514] Processing DeleteTablet for tablet d6b248113030427186ea796758fbbee1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:44:05 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:40886
I20260619 16:44:05.563233 8373 tablet_service.cc:1514] Processing DeleteTablet for tablet d6b248113030427186ea796758fbbee1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:44:05 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:60330
I20260619 16:44:05.563436 8538 tablet_replica.cc:330] T d6b248113030427186ea796758fbbee1 P f9432d0f6c60491b8a20b8dcf3561c84: stopping tablet replica
I20260619 16:44:05.563478 8540 tablet_replica.cc:330] T d6b248113030427186ea796758fbbee1 P 2ebc7147b38d42c183f6c7b6a62db075: stopping tablet replica
I20260619 16:44:05.563616 8538 raft_consensus.cc:2229] T d6b248113030427186ea796758fbbee1 P f9432d0f6c60491b8a20b8dcf3561c84 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 16:44:05.563643 8540 raft_consensus.cc:2229] T d6b248113030427186ea796758fbbee1 P 2ebc7147b38d42c183f6c7b6a62db075 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 16:44:05.563822 8540 raft_consensus.cc:2258] T d6b248113030427186ea796758fbbee1 P 2ebc7147b38d42c183f6c7b6a62db075 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:44:05.563827 8538 raft_consensus.cc:2258] T d6b248113030427186ea796758fbbee1 P f9432d0f6c60491b8a20b8dcf3561c84 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:44:05.564589 8235 tablet_service.cc:1514] Processing DeleteTablet for tablet d6b248113030427186ea796758fbbee1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:44:05 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:59360
I20260619 16:44:05.565079 8539 tablet_replica.cc:330] T d6b248113030427186ea796758fbbee1 P 410592fd61c042c9a956aada0e560a12: stopping tablet replica
I20260619 16:44:05.565207 8540 ts_tablet_manager.cc:1905] T d6b248113030427186ea796758fbbee1 P 2ebc7147b38d42c183f6c7b6a62db075: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:44:05.565279 8539 raft_consensus.cc:2229] T d6b248113030427186ea796758fbbee1 P 410592fd61c042c9a956aada0e560a12 [term 1 LEADER]: Raft consensus shutting down.
I20260619 16:44:05.565764 8539 raft_consensus.cc:2258] T d6b248113030427186ea796758fbbee1 P 410592fd61c042c9a956aada0e560a12 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:44:05.567404 8538 ts_tablet_manager.cc:1905] T d6b248113030427186ea796758fbbee1 P f9432d0f6c60491b8a20b8dcf3561c84: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:44:05.570353 8540 ts_tablet_manager.cc:1918] T d6b248113030427186ea796758fbbee1 P 2ebc7147b38d42c183f6c7b6a62db075: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 16:44:05.570449 8540 log.cc:1196] T d6b248113030427186ea796758fbbee1 P 2ebc7147b38d42c183f6c7b6a62db075: Deleting WAL directory at /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887385903558-388-0/minicluster-data/ts-2/wal/wals/d6b248113030427186ea796758fbbee1
I20260619 16:44:05.570701 8540 ts_tablet_manager.cc:1939] T d6b248113030427186ea796758fbbee1 P 2ebc7147b38d42c183f6c7b6a62db075: Deleting consensus metadata
I20260619 16:44:05.571570 7945 catalog_manager.cc:4841] TS 2ebc7147b38d42c183f6c7b6a62db075 (127.0.97.3:46789): tablet d6b248113030427186ea796758fbbee1 (table test-table [id=36dcae5d9e3a490abbcd32c42ee5dae5]) successfully deleted
I20260619 16:44:05.571720 7957 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48044:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 16:44:05.572353 7957 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 16:44:05.572616 8539 ts_tablet_manager.cc:1905] T d6b248113030427186ea796758fbbee1 P 410592fd61c042c9a956aada0e560a12: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:44:05.575366 8538 ts_tablet_manager.cc:1918] T d6b248113030427186ea796758fbbee1 P f9432d0f6c60491b8a20b8dcf3561c84: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 16:44:05.575575 8538 log.cc:1196] T d6b248113030427186ea796758fbbee1 P f9432d0f6c60491b8a20b8dcf3561c84: Deleting WAL directory at /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887385903558-388-0/minicluster-data/ts-0/wal/wals/d6b248113030427186ea796758fbbee1
I20260619 16:44:05.575990 8538 ts_tablet_manager.cc:1939] T d6b248113030427186ea796758fbbee1 P f9432d0f6c60491b8a20b8dcf3561c84: Deleting consensus metadata
I20260619 16:44:05.577062 7943 catalog_manager.cc:4841] TS f9432d0f6c60491b8a20b8dcf3561c84 (127.0.97.1:39607): tablet d6b248113030427186ea796758fbbee1 (table test-table [id=36dcae5d9e3a490abbcd32c42ee5dae5]) successfully deleted
I20260619 16:44:05.579290 8096 tablet_service.cc:1467] Processing CreateTablet for tablet 7537efc6ca21438580eec36f9c013e80 (DEFAULT_TABLE table=test-table [id=e2291cdf65654d34836af6e130c16d0d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:44:05.579704 8096 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7537efc6ca21438580eec36f9c013e80. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:44:05.579721 8373 tablet_service.cc:1467] Processing CreateTablet for tablet 7537efc6ca21438580eec36f9c013e80 (DEFAULT_TABLE table=test-table [id=e2291cdf65654d34836af6e130c16d0d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:44:05.579993 8539 ts_tablet_manager.cc:1918] T d6b248113030427186ea796758fbbee1 P 410592fd61c042c9a956aada0e560a12: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 16:44:05.580139 8373 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7537efc6ca21438580eec36f9c013e80. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:44:05.580133 8539 log.cc:1196] T d6b248113030427186ea796758fbbee1 P 410592fd61c042c9a956aada0e560a12: Deleting WAL directory at /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887385903558-388-0/minicluster-data/ts-1/wal/wals/d6b248113030427186ea796758fbbee1
I20260619 16:44:05.580529 8539 ts_tablet_manager.cc:1939] T d6b248113030427186ea796758fbbee1 P 410592fd61c042c9a956aada0e560a12: Deleting consensus metadata
I20260619 16:44:05.582027 8235 tablet_service.cc:1467] Processing CreateTablet for tablet 7537efc6ca21438580eec36f9c013e80 (DEFAULT_TABLE table=test-table [id=e2291cdf65654d34836af6e130c16d0d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:44:05.582404 8235 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7537efc6ca21438580eec36f9c013e80. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:44:05.585711 8470 tablet_bootstrap.cc:492] T 7537efc6ca21438580eec36f9c013e80 P 2ebc7147b38d42c183f6c7b6a62db075: Bootstrap starting.
I20260619 16:44:05.587216 8468 tablet_bootstrap.cc:492] T 7537efc6ca21438580eec36f9c013e80 P f9432d0f6c60491b8a20b8dcf3561c84: Bootstrap starting.
I20260619 16:44:05.587241 8470 tablet_bootstrap.cc:654] T 7537efc6ca21438580eec36f9c013e80 P 2ebc7147b38d42c183f6c7b6a62db075: Neither blocks nor log segments found. Creating new log.
I20260619 16:44:05.588582 7945 catalog_manager.cc:4841] TS 410592fd61c042c9a956aada0e560a12 (127.0.97.2:45795): tablet d6b248113030427186ea796758fbbee1 (table test-table [id=36dcae5d9e3a490abbcd32c42ee5dae5]) successfully deleted
I20260619 16:44:05.588748 8468 tablet_bootstrap.cc:654] T 7537efc6ca21438580eec36f9c013e80 P f9432d0f6c60491b8a20b8dcf3561c84: Neither blocks nor log segments found. Creating new log.
I20260619 16:44:05.589535 8471 tablet_bootstrap.cc:492] T 7537efc6ca21438580eec36f9c013e80 P 410592fd61c042c9a956aada0e560a12: Bootstrap starting.
I20260619 16:44:05.590220 8470 tablet_bootstrap.cc:492] T 7537efc6ca21438580eec36f9c013e80 P 2ebc7147b38d42c183f6c7b6a62db075: No bootstrap required, opened a new log
I20260619 16:44:05.590360 8470 ts_tablet_manager.cc:1397] T 7537efc6ca21438580eec36f9c013e80 P 2ebc7147b38d42c183f6c7b6a62db075: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.004s
I20260619 16:44:05.591140 8471 tablet_bootstrap.cc:654] T 7537efc6ca21438580eec36f9c013e80 P 410592fd61c042c9a956aada0e560a12: Neither blocks nor log segments found. Creating new log.
I20260619 16:44:05.591050 8470 raft_consensus.cc:348] T 7537efc6ca21438580eec36f9c013e80 P 2ebc7147b38d42c183f6c7b6a62db075 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9432d0f6c60491b8a20b8dcf3561c84" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39607 } } peers { permanent_uuid: "410592fd61c042c9a956aada0e560a12" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45795 } } peers { permanent_uuid: "2ebc7147b38d42c183f6c7b6a62db075" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46789 } }
I20260619 16:44:05.591271 8470 raft_consensus.cc:374] T 7537efc6ca21438580eec36f9c013e80 P 2ebc7147b38d42c183f6c7b6a62db075 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:44:05.591337 8470 raft_consensus.cc:729] T 7537efc6ca21438580eec36f9c013e80 P 2ebc7147b38d42c183f6c7b6a62db075 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ebc7147b38d42c183f6c7b6a62db075, State: Initialized, Role: FOLLOWER
I20260619 16:44:05.591588 8470 consensus_queue.cc:260] T 7537efc6ca21438580eec36f9c013e80 P 2ebc7147b38d42c183f6c7b6a62db075 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9432d0f6c60491b8a20b8dcf3561c84" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39607 } } peers { permanent_uuid: "410592fd61c042c9a956aada0e560a12" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45795 } } peers { permanent_uuid: "2ebc7147b38d42c183f6c7b6a62db075" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46789 } }
I20260619 16:44:05.594533 8470 ts_tablet_manager.cc:1428] T 7537efc6ca21438580eec36f9c013e80 P 2ebc7147b38d42c183f6c7b6a62db075: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20260619 16:44:05.598971 8468 tablet_bootstrap.cc:492] T 7537efc6ca21438580eec36f9c013e80 P f9432d0f6c60491b8a20b8dcf3561c84: No bootstrap required, opened a new log
I20260619 16:44:05.599103 8471 tablet_bootstrap.cc:492] T 7537efc6ca21438580eec36f9c013e80 P 410592fd61c042c9a956aada0e560a12: No bootstrap required, opened a new log
I20260619 16:44:05.599128 8468 ts_tablet_manager.cc:1397] T 7537efc6ca21438580eec36f9c013e80 P f9432d0f6c60491b8a20b8dcf3561c84: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.007s
I20260619 16:44:05.599267 8471 ts_tablet_manager.cc:1397] T 7537efc6ca21438580eec36f9c013e80 P 410592fd61c042c9a956aada0e560a12: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.007s
I20260619 16:44:05.600052 8471 raft_consensus.cc:348] T 7537efc6ca21438580eec36f9c013e80 P 410592fd61c042c9a956aada0e560a12 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9432d0f6c60491b8a20b8dcf3561c84" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39607 } } peers { permanent_uuid: "410592fd61c042c9a956aada0e560a12" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45795 } } peers { permanent_uuid: "2ebc7147b38d42c183f6c7b6a62db075" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46789 } }
I20260619 16:44:05.600145 8468 raft_consensus.cc:348] T 7537efc6ca21438580eec36f9c013e80 P f9432d0f6c60491b8a20b8dcf3561c84 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9432d0f6c60491b8a20b8dcf3561c84" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39607 } } peers { permanent_uuid: "410592fd61c042c9a956aada0e560a12" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45795 } } peers { permanent_uuid: "2ebc7147b38d42c183f6c7b6a62db075" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46789 } }
I20260619 16:44:05.600309 8471 raft_consensus.cc:374] T 7537efc6ca21438580eec36f9c013e80 P 410592fd61c042c9a956aada0e560a12 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:44:05.600373 8468 raft_consensus.cc:374] T 7537efc6ca21438580eec36f9c013e80 P f9432d0f6c60491b8a20b8dcf3561c84 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:44:05.600392 8471 raft_consensus.cc:729] T 7537efc6ca21438580eec36f9c013e80 P 410592fd61c042c9a956aada0e560a12 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 410592fd61c042c9a956aada0e560a12, State: Initialized, Role: FOLLOWER
I20260619 16:44:05.600450 8468 raft_consensus.cc:729] T 7537efc6ca21438580eec36f9c013e80 P f9432d0f6c60491b8a20b8dcf3561c84 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f9432d0f6c60491b8a20b8dcf3561c84, State: Initialized, Role: FOLLOWER
I20260619 16:44:05.600689 8468 consensus_queue.cc:260] T 7537efc6ca21438580eec36f9c013e80 P f9432d0f6c60491b8a20b8dcf3561c84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9432d0f6c60491b8a20b8dcf3561c84" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39607 } } peers { permanent_uuid: "410592fd61c042c9a956aada0e560a12" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45795 } } peers { permanent_uuid: "2ebc7147b38d42c183f6c7b6a62db075" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46789 } }
I20260619 16:44:05.600630 8471 consensus_queue.cc:260] T 7537efc6ca21438580eec36f9c013e80 P 410592fd61c042c9a956aada0e560a12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9432d0f6c60491b8a20b8dcf3561c84" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39607 } } peers { permanent_uuid: "410592fd61c042c9a956aada0e560a12" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45795 } } peers { permanent_uuid: "2ebc7147b38d42c183f6c7b6a62db075" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46789 } }
I20260619 16:44:05.601135 8471 ts_tablet_manager.cc:1428] T 7537efc6ca21438580eec36f9c013e80 P 410592fd61c042c9a956aada0e560a12: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260619 16:44:05.601222 8468 ts_tablet_manager.cc:1428] T 7537efc6ca21438580eec36f9c013e80 P f9432d0f6c60491b8a20b8dcf3561c84: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260619 16:44:05.723790 8533 raft_consensus.cc:482] T 7537efc6ca21438580eec36f9c013e80 P f9432d0f6c60491b8a20b8dcf3561c84 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 16:44:05.723959 8533 raft_consensus.cc:504] T 7537efc6ca21438580eec36f9c013e80 P f9432d0f6c60491b8a20b8dcf3561c84 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9432d0f6c60491b8a20b8dcf3561c84" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39607 } } peers { permanent_uuid: "410592fd61c042c9a956aada0e560a12" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45795 } } peers { permanent_uuid: "2ebc7147b38d42c183f6c7b6a62db075" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46789 } }
I20260619 16:44:05.724591 8533 leader_election.cc:288] T 7537efc6ca21438580eec36f9c013e80 P f9432d0f6c60491b8a20b8dcf3561c84 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 410592fd61c042c9a956aada0e560a12 (127.0.97.2:45795), 2ebc7147b38d42c183f6c7b6a62db075 (127.0.97.3:46789)
I20260619 16:44:05.725126 8254 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7537efc6ca21438580eec36f9c013e80" candidate_uuid: "f9432d0f6c60491b8a20b8dcf3561c84" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "410592fd61c042c9a956aada0e560a12" is_pre_election: true
I20260619 16:44:05.725220 8394 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7537efc6ca21438580eec36f9c013e80" candidate_uuid: "f9432d0f6c60491b8a20b8dcf3561c84" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ebc7147b38d42c183f6c7b6a62db075" is_pre_election: true
I20260619 16:44:05.725368 8254 raft_consensus.cc:2454] T 7537efc6ca21438580eec36f9c013e80 P 410592fd61c042c9a956aada0e560a12 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f9432d0f6c60491b8a20b8dcf3561c84 in term 0.
I20260619 16:44:05.725502 8394 raft_consensus.cc:2454] T 7537efc6ca21438580eec36f9c013e80 P 2ebc7147b38d42c183f6c7b6a62db075 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f9432d0f6c60491b8a20b8dcf3561c84 in term 0.
I20260619 16:44:05.725826 8048 leader_election.cc:302] T 7537efc6ca21438580eec36f9c013e80 P f9432d0f6c60491b8a20b8dcf3561c84 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 410592fd61c042c9a956aada0e560a12, f9432d0f6c60491b8a20b8dcf3561c84; no voters:
I20260619 16:44:05.726140 8533 raft_consensus.cc:2781] T 7537efc6ca21438580eec36f9c013e80 P f9432d0f6c60491b8a20b8dcf3561c84 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 16:44:05.726258 8533 raft_consensus.cc:482] T 7537efc6ca21438580eec36f9c013e80 P f9432d0f6c60491b8a20b8dcf3561c84 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 16:44:05.726346 8533 raft_consensus.cc:3037] T 7537efc6ca21438580eec36f9c013e80 P f9432d0f6c60491b8a20b8dcf3561c84 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:44:05.729121 8533 raft_consensus.cc:504] T 7537efc6ca21438580eec36f9c013e80 P f9432d0f6c60491b8a20b8dcf3561c84 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9432d0f6c60491b8a20b8dcf3561c84" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39607 } } peers { permanent_uuid: "410592fd61c042c9a956aada0e560a12" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45795 } } peers { permanent_uuid: "2ebc7147b38d42c183f6c7b6a62db075" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46789 } }
I20260619 16:44:05.729661 8533 leader_election.cc:288] T 7537efc6ca21438580eec36f9c013e80 P f9432d0f6c60491b8a20b8dcf3561c84 [CANDIDATE]: Term 1 election: Requested vote from peers 410592fd61c042c9a956aada0e560a12 (127.0.97.2:45795), 2ebc7147b38d42c183f6c7b6a62db075 (127.0.97.3:46789)
I20260619 16:44:05.730105 8254 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7537efc6ca21438580eec36f9c013e80" candidate_uuid: "f9432d0f6c60491b8a20b8dcf3561c84" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "410592fd61c042c9a956aada0e560a12"
I20260619 16:44:05.730159 8394 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7537efc6ca21438580eec36f9c013e80" candidate_uuid: "f9432d0f6c60491b8a20b8dcf3561c84" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ebc7147b38d42c183f6c7b6a62db075"
I20260619 16:44:05.730310 8254 raft_consensus.cc:3037] T 7537efc6ca21438580eec36f9c013e80 P 410592fd61c042c9a956aada0e560a12 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:44:05.730319 8394 raft_consensus.cc:3037] T 7537efc6ca21438580eec36f9c013e80 P 2ebc7147b38d42c183f6c7b6a62db075 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:44:05.732141 8394 raft_consensus.cc:2454] T 7537efc6ca21438580eec36f9c013e80 P 2ebc7147b38d42c183f6c7b6a62db075 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f9432d0f6c60491b8a20b8dcf3561c84 in term 1.
I20260619 16:44:05.732451 8048 leader_election.cc:302] T 7537efc6ca21438580eec36f9c013e80 P f9432d0f6c60491b8a20b8dcf3561c84 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2ebc7147b38d42c183f6c7b6a62db075, f9432d0f6c60491b8a20b8dcf3561c84; no voters:
I20260619 16:44:05.732698 8533 raft_consensus.cc:2781] T 7537efc6ca21438580eec36f9c013e80 P f9432d0f6c60491b8a20b8dcf3561c84 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:44:05.733068 8533 raft_consensus.cc:686] T 7537efc6ca21438580eec36f9c013e80 P f9432d0f6c60491b8a20b8dcf3561c84 [term 1 LEADER]: Becoming Leader. State: Replica: f9432d0f6c60491b8a20b8dcf3561c84, State: Running, Role: LEADER
I20260619 16:44:05.733222 8254 raft_consensus.cc:2454] T 7537efc6ca21438580eec36f9c013e80 P 410592fd61c042c9a956aada0e560a12 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f9432d0f6c60491b8a20b8dcf3561c84 in term 1.
I20260619 16:44:05.733382 8533 consensus_queue.cc:237] T 7537efc6ca21438580eec36f9c013e80 P f9432d0f6c60491b8a20b8dcf3561c84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9432d0f6c60491b8a20b8dcf3561c84" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39607 } } peers { permanent_uuid: "410592fd61c042c9a956aada0e560a12" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45795 } } peers { permanent_uuid: "2ebc7147b38d42c183f6c7b6a62db075" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46789 } }
I20260619 16:44:05.735788 7957 catalog_manager.cc:5495] T 7537efc6ca21438580eec36f9c013e80 P f9432d0f6c60491b8a20b8dcf3561c84 reported cstate change: term changed from 0 to 1, leader changed from <none> to f9432d0f6c60491b8a20b8dcf3561c84 (127.0.97.1). New cstate: current_term: 1 leader_uuid: "f9432d0f6c60491b8a20b8dcf3561c84" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9432d0f6c60491b8a20b8dcf3561c84" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39607 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "410592fd61c042c9a956aada0e560a12" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45795 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2ebc7147b38d42c183f6c7b6a62db075" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46789 } health_report { overall_health: UNKNOWN } } }
I20260619 16:44:05.759296 8511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:44:05.754386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40898 (local address 127.0.97.1:39607)
0619 16:44:05.754563 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0619 16:44:05.754569 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:44:05.754694 (+ 125us) server_negotiation.cc:407] Connection header received
0619 16:44:05.754856 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:44:05.754860 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:44:05.754926 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:44:05.754994 (+ 68us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 16:44:05.755331 (+ 337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:44:05.755984 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:44:05.757845 (+ 1861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:44:05.758221 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:44:05.758306 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:44:05.758730 (+ 424us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 16:44:05.758844 (+ 114us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 16:44:05.758904 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0619 16:44:05.759014 (+ 110us) server_negotiation.cc:299] Negotiation successful
0619 16:44:05.759069 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260619 16:44:05.767249 8394 raft_consensus.cc:1261] T 7537efc6ca21438580eec36f9c013e80 P 2ebc7147b38d42c183f6c7b6a62db075 [term 1 FOLLOWER]: Refusing update from remote peer f9432d0f6c60491b8a20b8dcf3561c84: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:44:05.767424 8255 raft_consensus.cc:1261] T 7537efc6ca21438580eec36f9c013e80 P 410592fd61c042c9a956aada0e560a12 [term 1 FOLLOWER]: Refusing update from remote peer f9432d0f6c60491b8a20b8dcf3561c84: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:44:05.767863 8533 consensus_queue.cc:1035] T 7537efc6ca21438580eec36f9c013e80 P f9432d0f6c60491b8a20b8dcf3561c84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ebc7147b38d42c183f6c7b6a62db075" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:44:05.768242 8574 consensus_queue.cc:1035] T 7537efc6ca21438580eec36f9c013e80 P f9432d0f6c60491b8a20b8dcf3561c84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "410592fd61c042c9a956aada0e560a12" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:44:05.817440 7957 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48044:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:44:05.817623 7957 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48044:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:44:05.820361 7957 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ed5a208fb1504c2dbce0eceab08ad43c: Sending DeleteTablet for 3 replicas of tablet 7537efc6ca21438580eec36f9c013e80
I20260619 16:44:05.821089 8096 tablet_service.cc:1514] Processing DeleteTablet for tablet 7537efc6ca21438580eec36f9c013e80 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:44:05 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:40886
I20260619 16:44:05.821368 8235 tablet_service.cc:1514] Processing DeleteTablet for tablet 7537efc6ca21438580eec36f9c013e80 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:44:05 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:59360
I20260619 16:44:05.821674 8538 tablet_replica.cc:330] T 7537efc6ca21438580eec36f9c013e80 P f9432d0f6c60491b8a20b8dcf3561c84: stopping tablet replica
I20260619 16:44:05.821880 8538 raft_consensus.cc:2229] T 7537efc6ca21438580eec36f9c013e80 P f9432d0f6c60491b8a20b8dcf3561c84 [term 1 LEADER]: Raft consensus shutting down.
I20260619 16:44:05.822217 8538 raft_consensus.cc:2258] T 7537efc6ca21438580eec36f9c013e80 P f9432d0f6c60491b8a20b8dcf3561c84 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:44:05.822424 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu pid 8030
I20260619 16:44:05.822494 8539 tablet_replica.cc:330] T 7537efc6ca21438580eec36f9c013e80 P 410592fd61c042c9a956aada0e560a12: stopping tablet replica
I20260619 16:44:05.822649 8539 raft_consensus.cc:2229] T 7537efc6ca21438580eec36f9c013e80 P 410592fd61c042c9a956aada0e560a12 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 16:44:05.822788 8539 raft_consensus.cc:2258] T 7537efc6ca21438580eec36f9c013e80 P 410592fd61c042c9a956aada0e560a12 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:44:05.824471 8373 tablet_service.cc:1514] Processing DeleteTablet for tablet 7537efc6ca21438580eec36f9c013e80 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:44:05 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:60330
I20260619 16:44:05.824898 8539 ts_tablet_manager.cc:1905] T 7537efc6ca21438580eec36f9c013e80 P 410592fd61c042c9a956aada0e560a12: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:44:05.825059 8538 ts_tablet_manager.cc:1905] T 7537efc6ca21438580eec36f9c013e80 P f9432d0f6c60491b8a20b8dcf3561c84: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:44:05.829921 8540 tablet_replica.cc:330] T 7537efc6ca21438580eec36f9c013e80 P 2ebc7147b38d42c183f6c7b6a62db075: stopping tablet replica
I20260619 16:44:05.830108 8540 raft_consensus.cc:2229] T 7537efc6ca21438580eec36f9c013e80 P 2ebc7147b38d42c183f6c7b6a62db075 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 16:44:05.830307 8540 raft_consensus.cc:2258] T 7537efc6ca21438580eec36f9c013e80 P 2ebc7147b38d42c183f6c7b6a62db075 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:44:05.831394 8538 ts_tablet_manager.cc:1918] T 7537efc6ca21438580eec36f9c013e80 P f9432d0f6c60491b8a20b8dcf3561c84: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 16:44:05.831584 8538 log.cc:1196] T 7537efc6ca21438580eec36f9c013e80 P f9432d0f6c60491b8a20b8dcf3561c84: Deleting WAL directory at /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887385903558-388-0/minicluster-data/ts-0/wal/wals/7537efc6ca21438580eec36f9c013e80
I20260619 16:44:05.831590 8540 ts_tablet_manager.cc:1905] T 7537efc6ca21438580eec36f9c013e80 P 2ebc7147b38d42c183f6c7b6a62db075: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:44:05.832082 8538 ts_tablet_manager.cc:1939] T 7537efc6ca21438580eec36f9c013e80 P f9432d0f6c60491b8a20b8dcf3561c84: Deleting consensus metadata
I20260619 16:44:05.833065 7943 catalog_manager.cc:4841] TS f9432d0f6c60491b8a20b8dcf3561c84 (127.0.97.1:39607): tablet 7537efc6ca21438580eec36f9c013e80 (table test-table [id=e2291cdf65654d34836af6e130c16d0d]) successfully deleted
I20260619 16:44:05.839843 8539 ts_tablet_manager.cc:1918] T 7537efc6ca21438580eec36f9c013e80 P 410592fd61c042c9a956aada0e560a12: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 16:44:05.840055 8539 log.cc:1196] T 7537efc6ca21438580eec36f9c013e80 P 410592fd61c042c9a956aada0e560a12: Deleting WAL directory at /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887385903558-388-0/minicluster-data/ts-1/wal/wals/7537efc6ca21438580eec36f9c013e80
I20260619 16:44:05.840494 8539 ts_tablet_manager.cc:1939] T 7537efc6ca21438580eec36f9c013e80 P 410592fd61c042c9a956aada0e560a12: Deleting consensus metadata
I20260619 16:44:05.841673 7945 catalog_manager.cc:4841] TS 410592fd61c042c9a956aada0e560a12 (127.0.97.2:45795): tablet 7537efc6ca21438580eec36f9c013e80 (table test-table [id=e2291cdf65654d34836af6e130c16d0d]) successfully deleted
I20260619 16:44:05.842039 8540 ts_tablet_manager.cc:1918] T 7537efc6ca21438580eec36f9c013e80 P 2ebc7147b38d42c183f6c7b6a62db075: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 16:44:05.842185 8540 log.cc:1196] T 7537efc6ca21438580eec36f9c013e80 P 2ebc7147b38d42c183f6c7b6a62db075: Deleting WAL directory at /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887385903558-388-0/minicluster-data/ts-2/wal/wals/7537efc6ca21438580eec36f9c013e80
I20260619 16:44:05.842583 8540 ts_tablet_manager.cc:1939] T 7537efc6ca21438580eec36f9c013e80 P 2ebc7147b38d42c183f6c7b6a62db075: Deleting consensus metadata
Jun 19 16:44:05 dist-test-slave-wvz7 krb5kdc[7914](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887445, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.1@KRBTEST.COM
I20260619 16:44:05.843719 7945 catalog_manager.cc:4841] TS 2ebc7147b38d42c183f6c7b6a62db075 (127.0.97.3:46789): tablet 7537efc6ca21438580eec36f9c013e80 (table test-table [id=e2291cdf65654d34836af6e130c16d0d]) successfully deleted
I20260619 16:44:05.849799 8511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:44:05.823527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40904 (local address 127.0.97.1:39607)
0619 16:44:05.823671 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0619 16:44:05.823678 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:44:05.834161 (+ 10483us) server_negotiation.cc:407] Connection header received
0619 16:44:05.834277 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:44:05.834282 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:44:05.834360 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:44:05.834467 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:44:05.836090 (+ 1623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:44:05.837118 (+ 1028us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:44:05.839015 (+ 1897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:44:05.839634 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:44:05.839727 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:44:05.844537 (+ 4810us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:44:05.844551 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:44:05.844565 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:44:05.844599 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:44:05.847303 (+ 2704us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:44:05.847933 (+ 630us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:44:05.847942 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:44:05.847946 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:44:05.848075 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:44:05.848415 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:44:05.848420 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:44:05.848422 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:44:05.848744 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:44:05.849075 (+ 331us) server_negotiation.cc:1035] Waiting for connection context
0619 16:44:05.849250 (+ 175us) server_negotiation.cc:299] Negotiation successful
0619 16:44:05.849446 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260619 16:44:05.850346 8161 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:44:06.018327 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu with pid 8030
I20260619 16:44:06.036124 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu pid 8170
Jun 19 16:44:06 dist-test-slave-wvz7 krb5kdc[7914](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887445, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.2@KRBTEST.COM
I20260619 16:44:06.051182 8585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:44:06.036868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59386 (local address 127.0.97.2:45795)
0619 16:44:06.037329 (+ 461us) server_negotiation.cc:206] Beginning negotiation
0619 16:44:06.037333 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:44:06.037349 (+ 16us) server_negotiation.cc:407] Connection header received
0619 16:44:06.037397 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:44:06.037400 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:44:06.037455 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:44:06.037541 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:44:06.038851 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:44:06.039624 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:44:06.040935 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:44:06.041327 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:44:06.041415 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:44:06.045496 (+ 4081us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:44:06.045508 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:44:06.045522 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:44:06.045552 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:44:06.048608 (+ 3056us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:44:06.049309 (+ 701us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:44:06.049316 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:44:06.049319 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:44:06.049463 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:44:06.049854 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:44:06.049858 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:44:06.049860 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:44:06.050196 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:44:06.050452 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0619 16:44:06.050734 (+ 282us) server_negotiation.cc:299] Negotiation successful
0619 16:44:06.050924 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":374,"thread_start_us":252,"threads_started":1}
I20260619 16:44:06.051785 8300 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:44:06.168388 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu with pid 8170
I20260619 16:44:06.185685 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu pid 8309
Jun 19 16:44:06 dist-test-slave-wvz7 krb5kdc[7914](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887445, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.3@KRBTEST.COM
I20260619 16:44:06.199533 8587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:44:06.186385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60350 (local address 127.0.97.3:46789)
0619 16:44:06.186822 (+ 437us) server_negotiation.cc:206] Beginning negotiation
0619 16:44:06.186826 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:44:06.186841 (+ 15us) server_negotiation.cc:407] Connection header received
0619 16:44:06.186893 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:44:06.186896 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:44:06.186959 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:44:06.187044 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:44:06.188182 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:44:06.188884 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:44:06.190195 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:44:06.190604 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:44:06.190700 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:44:06.194691 (+ 3991us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:44:06.194706 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:44:06.194721 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:44:06.194752 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:44:06.197276 (+ 2524us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:44:06.197917 (+ 641us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:44:06.197923 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:44:06.197925 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:44:06.198036 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:44:06.198329 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:44:06.198333 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:44:06.198335 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:44:06.198666 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:44:06.198902 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0619 16:44:06.199066 (+ 164us) server_negotiation.cc:299] Negotiation successful
0619 16:44:06.199219 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":318,"thread_start_us":222,"threads_started":1}
I20260619 16:44:06.200115 8439 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:44:06.316566 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu with pid 8309
I20260619 16:44:06.333022 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu pid 7927
I20260619 16:44:06.334224 7999 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:44:06.446158 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu with pid 7927
2026-06-19T16:44:06Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (5559 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781887385903558-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 16:44:06 dist-test-slave-wvz7 krb5kdc[8594](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 16:44:06 dist-test-slave-wvz7 krb5kdc[8594](info): set up 2 sockets
Jun 19 16:44:06 dist-test-slave-wvz7 krb5kdc[8594](info): commencing operation
krb5kdc: starting...
W20260619 16:44:07.442893 388 mini_kdc.cc:121] Time spent starting KDC: real 0.958s user 0.004s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781887385903558-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781887385903558-388-0/krb5kdc/test-user.keytab.
Jun 19 16:44:07 dist-test-slave-wvz7 krb5kdc[8594](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887447, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T16:44:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T16:44:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781887385903558-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781887385903558-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260619 16:44:07.628419 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu
/tmp/dist-test-taskEs2EH8/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781887385903558-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781887385903558-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781887385903558-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781887385903558-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:41399
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781887385903558-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44817
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:41399
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781887385903558-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781887385903558-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781887385903558-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:44:07.925928 8607 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:44:07.926188 8607 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:44:07.926298 8607 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:44:07.933205 8607 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:44:07.933305 8607 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:44:07.933372 8607 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:44:07.933440 8607 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:44:07.933488 8607 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:44:07.943049 8607 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44817
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781887385903558-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781887385903558-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:41399
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781887385903558-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:41399
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781887385903558-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781887385903558-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:44:07.944281 8607 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:44:07.945495 8607 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:44:07.955924 8612 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:44:07.955989 8615 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:44:07.955947 8613 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:44:07.956751 8607 server_base.cc:1034] running on GCE node
I20260619 16:44:07.957461 8607 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:44:07.958775 8607 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:44:07.959986 8607 hybrid_clock.cc:648] HybridClock initialized: now 1781887447959933 us; error 43 us; skew 500 ppm
I20260619 16:44:07.960314 8607 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-06-19T16:44:08Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1580 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887385903558-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 16:44:08 dist-test-slave-wvz7 krb5kdc[8621](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 16:44:08 dist-test-slave-wvz7 krb5kdc[8621](info): set up 2 sockets
Jun 19 16:44:08 dist-test-slave-wvz7 krb5kdc[8621](info): commencing operation
krb5kdc: starting...
W20260619 16:44:09.018344 388 mini_kdc.cc:121] Time spent starting KDC: real 0.954s user 0.000s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887385903558-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887385903558-388-0/krb5kdc/test-user.keytab.
Jun 19 16:44:09 dist-test-slave-wvz7 krb5kdc[8621](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887449, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T16:44:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T16:44:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887385903558-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887385903558-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260619 16:44:09.209165 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu
/tmp/dist-test-taskEs2EH8/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887385903558-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887385903558-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887385903558-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887385903558-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:46703
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887385903558-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34173
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:46703
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887385903558-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887385903558-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887385903558-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:44:09.514276 8634 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:44:09.514602 8634 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:44:09.514740 8634 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:44:09.521632 8634 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:44:09.521764 8634 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:44:09.521850 8634 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:44:09.521929 8634 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:44:09.521991 8634 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:44:09.531333 8634 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34173
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887385903558-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887385903558-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:46703
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887385903558-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:46703
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887385903558-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887385903558-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:44:09.532562 8634 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:44:09.533892 8634 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:44:09.544776 8639 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:44:09.544849 8642 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:44:09.544873 8640 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:44:09.545974 8634 server_base.cc:1034] running on GCE node
I20260619 16:44:09.546710 8634 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:44:09.547992 8634 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:44:09.549199 8634 hybrid_clock.cc:648] HybridClock initialized: now 1781887449549149 us; error 38 us; skew 500 ppm
I20260619 16:44:09.549522 8634 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:44:09 dist-test-slave-wvz7 krb5kdc[8621](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887449, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:44:09.552347 8634 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260619 16:44:09.553629 8634 webserver.cc:460] Webserver started at http://127.0.97.62:42401/ using document root <none> and password file <none>
I20260619 16:44:09.554373 8634 fs_manager.cc:359] Metadata directory not provided
I20260619 16:44:09.554487 8634 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:44:09.554724 8634 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:44:09.557425 8634 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887385903558-388-0/minicluster-data/master-0/data/instance:
uuid: "4055494c55824ee0b04372739fec3560"
format_stamp: "Formatted at 2026-06-19 16:44:09 on dist-test-slave-wvz7"
server_key: "2e0271742a81a58b3631cfa844416fae"
server_key_iv: "b4eedb7437c198bf02d14210256d6596"
server_key_version: "encryptionkey@0"
I20260619 16:44:09.558135 8634 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887385903558-388-0/minicluster-data/master-0/wal/instance:
uuid: "4055494c55824ee0b04372739fec3560"
format_stamp: "Formatted at 2026-06-19 16:44:09 on dist-test-slave-wvz7"
server_key: "2e0271742a81a58b3631cfa844416fae"
server_key_iv: "b4eedb7437c198bf02d14210256d6596"
server_key_version: "encryptionkey@0"
I20260619 16:44:09.562870 8634 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260619 16:44:09.566183 8649 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:44:09.567472 8634 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260619 16:44:09.567606 8634 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887385903558-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887385903558-388-0/minicluster-data/master-0/wal
uuid: "4055494c55824ee0b04372739fec3560"
format_stamp: "Formatted at 2026-06-19 16:44:09 on dist-test-slave-wvz7"
server_key: "2e0271742a81a58b3631cfa844416fae"
server_key_iv: "b4eedb7437c198bf02d14210256d6596"
server_key_version: "encryptionkey@0"
I20260619 16:44:09.567759 8634 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887385903558-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887385903558-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887385903558-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:44:09.596858 8634 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:44:09.600883 8634 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:44:09.601102 8634 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:44:09.616707 8634 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:46703
I20260619 16:44:09.616766 8700 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:46703 every 8 connection(s)
I20260619 16:44:09.617977 8634 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887385903558-388-0/minicluster-data/master-0/data/info.pb
I20260619 16:44:09.621410 8701 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:44:09.627156 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu as pid 8634
I20260619 16:44:09.627328 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887385903558-388-0/minicluster-data/master-0/wal/instance
I20260619 16:44:09.627885 388 external_mini_cluster.cc:1428] Setting key 04285b5e00ab8fa11c1be5826e6b4584
I20260619 16:44:09.631407 8701 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4055494c55824ee0b04372739fec3560: Bootstrap starting.
I20260619 16:44:09.634884 8701 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4055494c55824ee0b04372739fec3560: Neither blocks nor log segments found. Creating new log.
I20260619 16:44:09.636102 8701 log.cc:824] T 00000000000000000000000000000000 P 4055494c55824ee0b04372739fec3560: Log is configured to *not* fsync() on all Append() calls
I20260619 16:44:09.638917 8701 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4055494c55824ee0b04372739fec3560: No bootstrap required, opened a new log
Jun 19 16:44:09 dist-test-slave-wvz7 krb5kdc[8621](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887449, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 16:44:09.646274 8701 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4055494c55824ee0b04372739fec3560 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4055494c55824ee0b04372739fec3560" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46703 } }
I20260619 16:44:09.646678 8701 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4055494c55824ee0b04372739fec3560 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:44:09.646782 8701 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4055494c55824ee0b04372739fec3560 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4055494c55824ee0b04372739fec3560, State: Initialized, Role: FOLLOWER
I20260619 16:44:09.647368 8701 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4055494c55824ee0b04372739fec3560 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4055494c55824ee0b04372739fec3560" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46703 } }
I20260619 16:44:09.647598 8701 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4055494c55824ee0b04372739fec3560 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:44:09.647723 8701 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4055494c55824ee0b04372739fec3560 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:44:09.647897 8701 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4055494c55824ee0b04372739fec3560 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:44:09.649678 8704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:44:09.629665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57566 (local address 127.0.97.62:46703)
0619 16:44:09.630459 (+ 794us) server_negotiation.cc:206] Beginning negotiation
0619 16:44:09.630474 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 16:44:09.630525 (+ 51us) server_negotiation.cc:407] Connection header received
0619 16:44:09.632742 (+ 2217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:44:09.632768 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:44:09.633271 (+ 503us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:44:09.633675 (+ 404us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:44:09.634875 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:44:09.636757 (+ 1882us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:44:09.638094 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:44:09.639043 (+ 949us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:44:09.639221 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:44:09.642903 (+ 3682us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:44:09.642911 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:44:09.642928 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:44:09.642965 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:44:09.645761 (+ 2796us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:44:09.646559 (+ 798us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:44:09.646566 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:44:09.646571 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:44:09.646762 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:44:09.647178 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:44:09.647183 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:44:09.647187 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:44:09.647719 (+ 532us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:44:09.648007 (+ 288us) server_negotiation.cc:1035] Waiting for connection context
0619 16:44:09.648327 (+ 320us) server_negotiation.cc:299] Negotiation successful
0619 16:44:09.648640 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":529,"thread_start_us":286,"threads_started":1}
I20260619 16:44:09.650408 8701 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4055494c55824ee0b04372739fec3560 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4055494c55824ee0b04372739fec3560" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46703 } }
I20260619 16:44:09.650871 8701 leader_election.cc:302] T 00000000000000000000000000000000 P 4055494c55824ee0b04372739fec3560 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4055494c55824ee0b04372739fec3560; no voters:
I20260619 16:44:09.651481 8701 leader_election.cc:288] T 00000000000000000000000000000000 P 4055494c55824ee0b04372739fec3560 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 16:44:09.651561 8706 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4055494c55824ee0b04372739fec3560 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:44:09.652567 8706 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4055494c55824ee0b04372739fec3560 [term 1 LEADER]: Becoming Leader. State: Replica: 4055494c55824ee0b04372739fec3560, State: Running, Role: LEADER
I20260619 16:44:09.653056 8706 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4055494c55824ee0b04372739fec3560 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4055494c55824ee0b04372739fec3560" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46703 } }
I20260619 16:44:09.653484 8701 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4055494c55824ee0b04372739fec3560 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:44:09.655568 8708 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4055494c55824ee0b04372739fec3560 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4055494c55824ee0b04372739fec3560. Latest consensus state: current_term: 1 leader_uuid: "4055494c55824ee0b04372739fec3560" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4055494c55824ee0b04372739fec3560" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46703 } } }
I20260619 16:44:09.655929 8708 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4055494c55824ee0b04372739fec3560 [sys.catalog]: This master's current role is: LEADER
I20260619 16:44:09.656736 8707 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4055494c55824ee0b04372739fec3560 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4055494c55824ee0b04372739fec3560" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4055494c55824ee0b04372739fec3560" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46703 } } }
I20260619 16:44:09.656997 8707 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4055494c55824ee0b04372739fec3560 [sys.catalog]: This master's current role is: LEADER
I20260619 16:44:09.663205 8714 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:44:09.666891 8714 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:44:09.673698 8714 catalog_manager.cc:1347] Generated new cluster ID: b7c38bbacd6d4bb6826f2cf0a69372af
I20260619 16:44:09.673832 8714 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:44:09.689481 8714 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 16:44:09.690688 8714 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:44:09.704466 8714 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4055494c55824ee0b04372739fec3560: Generated new TSK 0
I20260619 16:44:09.705381 8714 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887385903558-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887385903558-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260619 16:44:09.757296 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu
/tmp/dist-test-taskEs2EH8/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887385903558-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887385903558-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887385903558-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887385903558-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46703
--keytab_file=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887385903558-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34173
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887385903558-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887385903558-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887385903558-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:44:10.063861 8727 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:44:10.064136 8727 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:44:10.064249 8727 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:44:10.071566 8727 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:44:10.071671 8727 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:44:10.071806 8727 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260619 16:44:10.081301 8727 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34173
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887385903558-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887385903558-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887385903558-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887385903558-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46703
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887385903558-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:44:10.082561 8727 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:44:10.083803 8727 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:44:10.095551 8735 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:44:10.095628 8733 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:44:10.095775 8732 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:44:10.096401 8727 server_base.cc:1034] running on GCE node
I20260619 16:44:10.096925 8727 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:44:10.097824 8727 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:44:10.099012 8727 hybrid_clock.cc:648] HybridClock initialized: now 1781887450098962 us; error 44 us; skew 500 ppm
I20260619 16:44:10.099313 8727 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:44:10 dist-test-slave-wvz7 krb5kdc[8621](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887450, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:44:10.102174 8727 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260619 16:44:10.103552 8727 webserver.cc:460] Webserver started at http://127.0.97.1:40037/ using document root <none> and password file <none>
I20260619 16:44:10.104195 8727 fs_manager.cc:359] Metadata directory not provided
I20260619 16:44:10.104292 8727 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:44:10.104573 8727 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:44:10.107220 8727 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887385903558-388-0/minicluster-data/ts-0/data/instance:
uuid: "42616ba1a2424d8188c6114cd093d7a9"
format_stamp: "Formatted at 2026-06-19 16:44:10 on dist-test-slave-wvz7"
server_key: "c87b9f2d076d34093fc9cbd89f052caf"
server_key_iv: "c4aaca8cdb729788830f124186ae864c"
server_key_version: "encryptionkey@0"
I20260619 16:44:10.107919 8727 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887385903558-388-0/minicluster-data/ts-0/wal/instance:
uuid: "42616ba1a2424d8188c6114cd093d7a9"
format_stamp: "Formatted at 2026-06-19 16:44:10 on dist-test-slave-wvz7"
server_key: "c87b9f2d076d34093fc9cbd89f052caf"
server_key_iv: "c4aaca8cdb729788830f124186ae864c"
server_key_version: "encryptionkey@0"
I20260619 16:44:10.112506 8727 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260619 16:44:10.115811 8742 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:44:10.117098 8727 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 16:44:10.117234 8727 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887385903558-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887385903558-388-0/minicluster-data/ts-0/wal
uuid: "42616ba1a2424d8188c6114cd093d7a9"
format_stamp: "Formatted at 2026-06-19 16:44:10 on dist-test-slave-wvz7"
server_key: "c87b9f2d076d34093fc9cbd89f052caf"
server_key_iv: "c4aaca8cdb729788830f124186ae864c"
server_key_version: "encryptionkey@0"
I20260619 16:44:10.117449 8727 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887385903558-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887385903558-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887385903558-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:44:10.139145 8727 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:44:10.142768 8727 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:44:10.142983 8727 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:44:10.143829 8727 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:44:10.145025 8727 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:44:10.145094 8727 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:44:10.145177 8727 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:44:10.145217 8727 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:44:10.173310 8727 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:36879
I20260619 16:44:10.173379 8854 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:36879 every 8 connection(s)
I20260619 16:44:10.174546 8727 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887385903558-388-0/minicluster-data/ts-0/data/info.pb
I20260619 16:44:10.175469 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu as pid 8727
I20260619 16:44:10.175649 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887385903558-388-0/minicluster-data/ts-0/wal/instance
I20260619 16:44:10.176226 388 external_mini_cluster.cc:1428] Setting key e251b5072d471e2315e3e1f2b52f0685
Jun 19 16:44:10 dist-test-slave-wvz7 krb5kdc[8621](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887450, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 16:44:10.193533 8857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:44:10.176946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:47011 (local address 127.0.97.62:46703)
0619 16:44:10.177384 (+ 438us) server_negotiation.cc:206] Beginning negotiation
0619 16:44:10.177389 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:44:10.178416 (+ 1027us) server_negotiation.cc:407] Connection header received
0619 16:44:10.180129 (+ 1713us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:44:10.180134 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:44:10.180212 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:44:10.180307 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:44:10.181710 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:44:10.182452 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:44:10.184356 (+ 1904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:44:10.184741 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:44:10.184847 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:44:10.188541 (+ 3694us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:44:10.188546 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:44:10.188549 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:44:10.188578 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:44:10.190805 (+ 2227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:44:10.191521 (+ 716us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:44:10.191526 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:44:10.191529 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:44:10.191633 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:44:10.191996 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:44:10.192001 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:44:10.192004 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:44:10.192257 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:44:10.192380 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0619 16:44:10.192994 (+ 614us) server_negotiation.cc:299] Negotiation successful
0619 16:44:10.193192 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":309,"thread_start_us":212,"threads_started":1}
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260619 16:44:10.194482 8858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:44:10.177505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46703 (local address 127.0.97.1:47011)
0619 16:44:10.178277 (+ 772us) negotiation.cc:107] Waiting for socket to connect
0619 16:44:10.178313 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0619 16:44:10.179887 (+ 1574us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:44:10.180534 (+ 647us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:44:10.180542 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:44:10.181029 (+ 487us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:44:10.181553 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:44:10.181574 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:44:10.182590 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:44:10.182594 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:44:10.184196 (+ 1602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:44:10.184211 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:44:10.184958 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:44:10.184965 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:44:10.185235 (+ 270us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:44:10.185904 (+ 669us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:44:10.185930 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:44:10.188345 (+ 2415us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:44:10.190935 (+ 2590us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:44:10.190943 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:44:10.190959 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:44:10.191420 (+ 461us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:44:10.191727 (+ 307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:44:10.191732 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:44:10.191736 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:44:10.191901 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:44:10.192376 (+ 475us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:44:10.192383 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:44:10.192676 (+ 293us) client_negotiation.cc:769] Sending connection context
0619 16:44:10.192910 (+ 234us) client_negotiation.cc:240] Negotiation successful
0619 16:44:10.193394 (+ 484us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":549,"thread_start_us":346,"threads_started":1}
I20260619 16:44:10.195971 8855 heartbeater.cc:347] Connected to a master server at 127.0.97.62:46703
I20260619 16:44:10.196292 8855 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260619 16:44:10.197067 8855 heartbeater.cc:511] Master 127.0.97.62:46703 requested a full tablet report, sending...
I20260619 16:44:10.199261 8666 ts_manager.cc:194] Registered new tserver with Master: 42616ba1a2424d8188c6114cd093d7a9 (127.0.97.1:36879)
I20260619 16:44:10.200745 8666 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:47011
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887385903558-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887385903558-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260619 16:44:10.218238 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu
/tmp/dist-test-taskEs2EH8/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887385903558-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887385903558-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887385903558-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887385903558-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46703
--keytab_file=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887385903558-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34173
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887385903558-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887385903558-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887385903558-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:44:10.519878 8861 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:44:10.520143 8861 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:44:10.520239 8861 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:44:10.527235 8861 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:44:10.527336 8861 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:44:10.527442 8861 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260619 16:44:10.536746 8861 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34173
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887385903558-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887385903558-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887385903558-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887385903558-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46703
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887385903558-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:44:10.537966 8861 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:44:10.539158 8861 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:44:10.551124 8866 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:44:10.551126 8867 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:44:10.551157 8869 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:44:10.551708 8861 server_base.cc:1034] running on GCE node
I20260619 16:44:10.552265 8861 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:44:10.553128 8861 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:44:10.554317 8861 hybrid_clock.cc:648] HybridClock initialized: now 1781887450554250 us; error 60 us; skew 500 ppm
I20260619 16:44:10.554631 8861 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:44:10 dist-test-slave-wvz7 krb5kdc[8621](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887450, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:44:10.557521 8861 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260619 16:44:10.559008 8861 webserver.cc:460] Webserver started at http://127.0.97.2:34321/ using document root <none> and password file <none>
I20260619 16:44:10.559633 8861 fs_manager.cc:359] Metadata directory not provided
I20260619 16:44:10.559739 8861 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:44:10.560019 8861 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:44:10.562716 8861 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887385903558-388-0/minicluster-data/ts-1/data/instance:
uuid: "df3833d59e3244a3b0ae9f68ee2d9c94"
format_stamp: "Formatted at 2026-06-19 16:44:10 on dist-test-slave-wvz7"
server_key: "9343e96794e99a2e89a08b898c14f75d"
server_key_iv: "3221070840d7a7cb1f314bef16c65951"
server_key_version: "encryptionkey@0"
I20260619 16:44:10.563453 8861 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887385903558-388-0/minicluster-data/ts-1/wal/instance:
uuid: "df3833d59e3244a3b0ae9f68ee2d9c94"
format_stamp: "Formatted at 2026-06-19 16:44:10 on dist-test-slave-wvz7"
server_key: "9343e96794e99a2e89a08b898c14f75d"
server_key_iv: "3221070840d7a7cb1f314bef16c65951"
server_key_version: "encryptionkey@0"
I20260619 16:44:10.568049 8861 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 16:44:10.571229 8876 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:44:10.572476 8861 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 16:44:10.572602 8861 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887385903558-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887385903558-388-0/minicluster-data/ts-1/wal
uuid: "df3833d59e3244a3b0ae9f68ee2d9c94"
format_stamp: "Formatted at 2026-06-19 16:44:10 on dist-test-slave-wvz7"
server_key: "9343e96794e99a2e89a08b898c14f75d"
server_key_iv: "3221070840d7a7cb1f314bef16c65951"
server_key_version: "encryptionkey@0"
I20260619 16:44:10.572764 8861 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887385903558-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887385903558-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887385903558-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:44:10.586045 8861 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:44:10.589630 8861 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:44:10.589893 8861 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:44:10.592823 8861 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:44:10.594620 8861 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:44:10.594758 8861 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.001s
I20260619 16:44:10.594918 8861 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:44:10.595016 8861 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:44:10.630728 8861 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:39025
I20260619 16:44:10.630760 8988 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:39025 every 8 connection(s)
I20260619 16:44:10.631974 8861 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887385903558-388-0/minicluster-data/ts-1/data/info.pb
I20260619 16:44:10.634428 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu as pid 8861
I20260619 16:44:10.634593 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887385903558-388-0/minicluster-data/ts-1/wal/instance
I20260619 16:44:10.635092 388 external_mini_cluster.cc:1428] Setting key b969c34dbec3b004a38aa1a3a63edd77
Jun 19 16:44:10 dist-test-slave-wvz7 krb5kdc[8621](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887450, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260619 16:44:10.656045 8857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:44:10.634457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:39517 (local address 127.0.97.62:46703)
0619 16:44:10.634640 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0619 16:44:10.634646 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:44:10.636123 (+ 1477us) server_negotiation.cc:407] Connection header received
0619 16:44:10.638775 (+ 2652us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:44:10.638780 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:44:10.638865 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:44:10.638964 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:44:10.640931 (+ 1967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:44:10.641915 (+ 984us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:44:10.644398 (+ 2483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:44:10.644989 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:44:10.645112 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:44:10.649340 (+ 4228us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:44:10.649348 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:44:10.649353 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:44:10.649394 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:44:10.652774 (+ 3380us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:44:10.653644 (+ 870us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:44:10.653649 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:44:10.653654 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:44:10.653843 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:44:10.654351 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:44:10.654356 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:44:10.654360 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:44:10.654669 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:44:10.654790 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0619 16:44:10.655492 (+ 702us) server_negotiation.cc:299] Negotiation successful
0619 16:44:10.655709 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
I20260619 16:44:10.657114 8991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:44:10.635086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46703 (local address 127.0.97.2:39517)
0619 16:44:10.635954 (+ 868us) negotiation.cc:107] Waiting for socket to connect
0619 16:44:10.636011 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0619 16:44:10.638448 (+ 2437us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:44:10.639214 (+ 766us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:44:10.639227 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:44:10.639960 (+ 733us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:44:10.640751 (+ 791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:44:10.640775 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:44:10.642082 (+ 1307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:44:10.642088 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:44:10.644203 (+ 2115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:44:10.644219 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:44:10.645208 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:44:10.645216 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:44:10.645509 (+ 293us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:44:10.646293 (+ 784us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:44:10.646325 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:44:10.649085 (+ 2760us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:44:10.652926 (+ 3841us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:44:10.652935 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:44:10.652953 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:44:10.653527 (+ 574us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:44:10.653958 (+ 431us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:44:10.653963 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:44:10.653967 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:44:10.654197 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:44:10.654813 (+ 616us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:44:10.654821 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:44:10.655138 (+ 317us) client_negotiation.cc:769] Sending connection context
0619 16:44:10.655374 (+ 236us) client_negotiation.cc:240] Negotiation successful
0619 16:44:10.655896 (+ 522us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":563,"thread_start_us":345,"threads_started":1}
I20260619 16:44:10.658583 8989 heartbeater.cc:347] Connected to a master server at 127.0.97.62:46703
I20260619 16:44:10.658869 8989 heartbeater.cc:464] Registering TS with master...
I20260619 16:44:10.659610 8989 heartbeater.cc:511] Master 127.0.97.62:46703 requested a full tablet report, sending...
I20260619 16:44:10.660892 8666 ts_manager.cc:194] Registered new tserver with Master: df3833d59e3244a3b0ae9f68ee2d9c94 (127.0.97.2:39025)
I20260619 16:44:10.661674 8666 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:39517
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887385903558-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887385903558-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260619 16:44:10.686980 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu
/tmp/dist-test-taskEs2EH8/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887385903558-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887385903558-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887385903558-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887385903558-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46703
--keytab_file=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887385903558-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34173
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887385903558-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887385903558-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887385903558-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:44:11.015442 8994 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:44:11.015717 8994 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:44:11.015806 8994 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:44:11.022959 8994 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:44:11.023069 8994 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:44:11.023180 8994 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260619 16:44:11.032706 8994 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34173
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887385903558-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887385903558-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887385903558-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887385903558-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46703
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887385903558-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:44:11.033968 8994 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:44:11.035212 8994 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:44:11.047396 8999 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:44:11.047472 9002 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:44:11.047573 9000 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:44:11.048823 8994 server_base.cc:1034] running on GCE node
I20260619 16:44:11.049369 8994 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:44:11.050335 8994 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:44:11.051534 8994 hybrid_clock.cc:648] HybridClock initialized: now 1781887451051489 us; error 41 us; skew 500 ppm
I20260619 16:44:11.051831 8994 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:44:11 dist-test-slave-wvz7 krb5kdc[8621](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887451, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:44:11.054952 8994 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260619 16:44:11.056327 8994 webserver.cc:460] Webserver started at http://127.0.97.3:40441/ using document root <none> and password file <none>
I20260619 16:44:11.056941 8994 fs_manager.cc:359] Metadata directory not provided
I20260619 16:44:11.057051 8994 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:44:11.057331 8994 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:44:11.060089 8994 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887385903558-388-0/minicluster-data/ts-2/data/instance:
uuid: "dfdff7e441fc4bdda1489ff74e8bfa74"
format_stamp: "Formatted at 2026-06-19 16:44:11 on dist-test-slave-wvz7"
server_key: "ebc37e54066dc04a52b0592106f10911"
server_key_iv: "f4986a8592bd1e36ff365e5136be9b38"
server_key_version: "encryptionkey@0"
I20260619 16:44:11.060792 8994 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887385903558-388-0/minicluster-data/ts-2/wal/instance:
uuid: "dfdff7e441fc4bdda1489ff74e8bfa74"
format_stamp: "Formatted at 2026-06-19 16:44:11 on dist-test-slave-wvz7"
server_key: "ebc37e54066dc04a52b0592106f10911"
server_key_iv: "f4986a8592bd1e36ff365e5136be9b38"
server_key_version: "encryptionkey@0"
I20260619 16:44:11.065537 8994 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.002s
I20260619 16:44:11.068751 9009 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:44:11.070024 8994 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 16:44:11.070154 8994 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887385903558-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887385903558-388-0/minicluster-data/ts-2/wal
uuid: "dfdff7e441fc4bdda1489ff74e8bfa74"
format_stamp: "Formatted at 2026-06-19 16:44:11 on dist-test-slave-wvz7"
server_key: "ebc37e54066dc04a52b0592106f10911"
server_key_iv: "f4986a8592bd1e36ff365e5136be9b38"
server_key_version: "encryptionkey@0"
I20260619 16:44:11.070328 8994 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887385903558-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887385903558-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887385903558-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:44:11.103749 8994 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:44:11.107597 8994 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:44:11.107849 8994 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:44:11.108757 8994 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:44:11.110042 8994 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:44:11.110126 8994 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:44:11.110244 8994 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:44:11.110294 8994 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:44:11.139453 8994 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:45155
I20260619 16:44:11.139495 9121 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:45155 every 8 connection(s)
I20260619 16:44:11.140635 8994 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887385903558-388-0/minicluster-data/ts-2/data/info.pb
I20260619 16:44:11.149478 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu as pid 8994
I20260619 16:44:11.149631 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887385903558-388-0/minicluster-data/ts-2/wal/instance
I20260619 16:44:11.150133 388 external_mini_cluster.cc:1428] Setting key c1e9547e2c47ea60789a730b2cdb233b
Jun 19 16:44:11 dist-test-slave-wvz7 krb5kdc[8621](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887451, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 16:44:11.158774 8857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:44:11.142943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:55439 (local address 127.0.97.62:46703)
0619 16:44:11.143104 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0619 16:44:11.143108 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:44:11.144228 (+ 1120us) server_negotiation.cc:407] Connection header received
0619 16:44:11.145977 (+ 1749us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:44:11.145982 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:44:11.146061 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:44:11.146140 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:44:11.147520 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:44:11.148256 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:44:11.149990 (+ 1734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:44:11.150416 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:44:11.150514 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:44:11.153875 (+ 3361us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:44:11.153879 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:44:11.153882 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:44:11.153913 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:44:11.156274 (+ 2361us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:44:11.156872 (+ 598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:44:11.156875 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:44:11.156878 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:44:11.156979 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:44:11.157351 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:44:11.157354 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:44:11.157356 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:44:11.157586 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:44:11.157684 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0619 16:44:11.158380 (+ 696us) server_negotiation.cc:299] Negotiation successful
0619 16:44:11.158546 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260619 16:44:11.159708 9124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:44:11.143452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46703 (local address 127.0.97.3:55439)
0619 16:44:11.144088 (+ 636us) negotiation.cc:107] Waiting for socket to connect
0619 16:44:11.144123 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0619 16:44:11.145691 (+ 1568us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:44:11.146373 (+ 682us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:44:11.146381 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:44:11.146846 (+ 465us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:44:11.147372 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:44:11.147388 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:44:11.148394 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:44:11.148398 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:44:11.149863 (+ 1465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:44:11.149877 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:44:11.150609 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:44:11.150615 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:44:11.150822 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:44:11.151476 (+ 654us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:44:11.151501 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:44:11.153666 (+ 2165us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:44:11.156383 (+ 2717us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:44:11.156390 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:44:11.156400 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:44:11.156767 (+ 367us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:44:11.157083 (+ 316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:44:11.157086 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:44:11.157089 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:44:11.157268 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:44:11.157694 (+ 426us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:44:11.157701 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:44:11.158070 (+ 369us) client_negotiation.cc:769] Sending connection context
0619 16:44:11.158290 (+ 220us) client_negotiation.cc:240] Negotiation successful
0619 16:44:11.158665 (+ 375us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":422,"thread_start_us":249,"threads_started":1}
I20260619 16:44:11.160966 9122 heartbeater.cc:347] Connected to a master server at 127.0.97.62:46703
I20260619 16:44:11.161244 9122 heartbeater.cc:464] Registering TS with master...
I20260619 16:44:11.161964 9122 heartbeater.cc:511] Master 127.0.97.62:46703 requested a full tablet report, sending...
I20260619 16:44:11.163255 8666 ts_manager.cc:194] Registered new tserver with Master: dfdff7e441fc4bdda1489ff74e8bfa74 (127.0.97.3:45155)
I20260619 16:44:11.164219 8666 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:55439
I20260619 16:44:11.164976 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887385903558-388-0/krb5kdc/oryx_127.0.97.1.keytab.
Entry for principal oryx/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887385903558-388-0/krb5kdc/oryx_127.0.97.1.keytab.
I20260619 16:44:11.203982 8855 heartbeater.cc:503] Master 127.0.97.62:46703 was elected leader, sending a full tablet report...
WARNING: no policy specified for oryx/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887385903558-388-0/krb5kdc/oryx_127.0.97.2.keytab.
Entry for principal oryx/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887385903558-388-0/krb5kdc/oryx_127.0.97.2.keytab.
WARNING: no policy specified for oryx/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887385903558-388-0/krb5kdc/oryx_127.0.97.3.keytab.
Entry for principal oryx/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887385903558-388-0/krb5kdc/oryx_127.0.97.3.keytab.
I20260619 16:44:11.273705 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu pid 8727
Jun 19 16:44:11 dist-test-slave-wvz7 krb5kdc[8621](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887449, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260619 16:44:11.288684 9132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:44:11.274904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47328 (local address 127.0.97.1:36879)
0619 16:44:11.275520 (+ 616us) server_negotiation.cc:206] Beginning negotiation
0619 16:44:11.275527 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:44:11.275553 (+ 26us) server_negotiation.cc:407] Connection header received
0619 16:44:11.275611 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:44:11.275618 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:44:11.275860 (+ 242us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:44:11.276031 (+ 171us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:44:11.277294 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:44:11.278316 (+ 1022us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:44:11.279497 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:44:11.280038 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:44:11.280129 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:44:11.283754 (+ 3625us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:44:11.283763 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:44:11.283778 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:44:11.283807 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:44:11.286348 (+ 2541us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:44:11.286964 (+ 616us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:44:11.286971 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:44:11.286973 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:44:11.287088 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:44:11.287368 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:44:11.287372 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:44:11.287374 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:44:11.287673 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:44:11.287880 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0619 16:44:11.288176 (+ 296us) server_negotiation.cc:299] Negotiation successful
0619 16:44:11.288345 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":477,"thread_start_us":319,"threads_started":1}
I20260619 16:44:11.289888 8850 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:44:11.368386 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu with pid 8727
I20260619 16:44:11.380949 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu pid 8861
Jun 19 16:44:11 dist-test-slave-wvz7 krb5kdc[8621](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887449, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260619 16:44:11.395357 9134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:44:11.381729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50064 (local address 127.0.97.2:39025)
0619 16:44:11.382308 (+ 579us) server_negotiation.cc:206] Beginning negotiation
0619 16:44:11.382315 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:44:11.382339 (+ 24us) server_negotiation.cc:407] Connection header received
0619 16:44:11.382395 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:44:11.382402 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:44:11.382601 (+ 199us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:44:11.382724 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:44:11.383863 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:44:11.384844 (+ 981us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:44:11.386098 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:44:11.386624 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:44:11.386719 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:44:11.390522 (+ 3803us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:44:11.390533 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:44:11.390545 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:44:11.390575 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:44:11.392979 (+ 2404us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:44:11.393540 (+ 561us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:44:11.393546 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:44:11.393549 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:44:11.393671 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:44:11.394151 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:44:11.394154 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:44:11.394156 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:44:11.394465 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:44:11.394673 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0619 16:44:11.394911 (+ 238us) server_negotiation.cc:299] Negotiation successful
0619 16:44:11.395089 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":425,"thread_start_us":282,"threads_started":1}
I20260619 16:44:11.396502 8984 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:44:11.476958 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu with pid 8861
I20260619 16:44:11.488662 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu pid 8994
Jun 19 16:44:11 dist-test-slave-wvz7 krb5kdc[8621](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887449, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260619 16:44:11.503785 9136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:44:11.489557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41242 (local address 127.0.97.3:45155)
0619 16:44:11.490277 (+ 720us) server_negotiation.cc:206] Beginning negotiation
0619 16:44:11.490283 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:44:11.490315 (+ 32us) server_negotiation.cc:407] Connection header received
0619 16:44:11.490385 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:44:11.490393 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:44:11.490602 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:44:11.490761 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:44:11.491917 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:44:11.492954 (+ 1037us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:44:11.494218 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:44:11.494814 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:44:11.494911 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:44:11.498833 (+ 3922us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:44:11.498842 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:44:11.498856 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:44:11.498888 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:44:11.501503 (+ 2615us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:44:11.502160 (+ 657us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:44:11.502168 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:44:11.502171 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:44:11.502320 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:44:11.502588 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:44:11.502591 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:44:11.502593 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:44:11.502902 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:44:11.503131 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0619 16:44:11.503332 (+ 201us) server_negotiation.cc:299] Negotiation successful
0619 16:44:11.503522 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":560,"thread_start_us":384,"threads_started":1}
I20260619 16:44:11.504964 9117 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:44:11.584434 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu with pid 8994
I20260619 16:44:11.596349 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu pid 8634
I20260619 16:44:11.597468 8696 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:44:11.694521 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu with pid 8634
I20260619 16:44:11.710359 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu
/tmp/dist-test-taskEs2EH8/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887385903558-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887385903558-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887385903558-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887385903558-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:46703
--webserver_interface=127.0.97.62
--webserver_port=42401
--keytab_file=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887385903558-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34173
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:46703
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887385903558-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887385903558-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887385903558-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:44:12.014424 9139 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:44:12.014695 9139 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:44:12.014832 9139 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:44:12.023753 9139 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:44:12.023881 9139 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:44:12.023959 9139 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:44:12.024029 9139 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:44:12.024087 9139 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:44:12.033818 9139 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34173
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887385903558-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887385903558-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:46703
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887385903558-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:46703
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887385903558-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=42401
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887385903558-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:44:12.035109 9139 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:44:12.036435 9139 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:44:12.047200 9144 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:44:12.047246 9147 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:44:12.047298 9145 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:44:12.048435 9139 server_base.cc:1034] running on GCE node
I20260619 16:44:12.049172 9139 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:44:12.050556 9139 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:44:12.051767 9139 hybrid_clock.cc:648] HybridClock initialized: now 1781887452051689 us; error 58 us; skew 500 ppm
I20260619 16:44:12.052183 9139 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:44:12 dist-test-slave-wvz7 krb5kdc[8621](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887452, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:44:12.056056 9139 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260619 16:44:12.057889 9139 webserver.cc:460] Webserver started at http://127.0.97.62:42401/ using document root <none> and password file <none>
I20260619 16:44:12.058730 9139 fs_manager.cc:359] Metadata directory not provided
I20260619 16:44:12.058894 9139 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:44:12.066021 9139 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.007s sys 0.000s
I20260619 16:44:12.069468 9154 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:44:12.070848 9139 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 16:44:12.071004 9139 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887385903558-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887385903558-388-0/minicluster-data/master-0/wal
uuid: "4055494c55824ee0b04372739fec3560"
format_stamp: "Formatted at 2026-06-19 16:44:09 on dist-test-slave-wvz7"
server_key: "2e0271742a81a58b3631cfa844416fae"
server_key_iv: "b4eedb7437c198bf02d14210256d6596"
server_key_version: "encryptionkey@0"
I20260619 16:44:12.071915 9139 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887385903558-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887385903558-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887385903558-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:44:12.089437 9139 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:44:12.093693 9139 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:44:12.093981 9139 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:44:12.110090 9139 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:46703
I20260619 16:44:12.110097 9205 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:46703 every 8 connection(s)
I20260619 16:44:12.111434 9139 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887385903558-388-0/minicluster-data/master-0/data/info.pb
I20260619 16:44:12.117811 9206 sys_catalog.cc:263] Verifying existing consensus state
I20260619 16:44:12.117935 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu as pid 9139
I20260619 16:44:12.118829 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu
/tmp/dist-test-taskEs2EH8/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887385903558-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887385903558-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887385903558-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887385903558-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:36879
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:46703
--webserver_port=40037
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887385903558-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34173
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.97.1
--keytab_file=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887385903558-388-0/krb5kdc/oryx_127.0.97.1.keytab with env {KRB5CCNAME=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887385903558-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887385903558-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887385903558-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 16:44:12.122079 9206 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4055494c55824ee0b04372739fec3560: Bootstrap starting.
I20260619 16:44:12.140621 9206 log.cc:824] T 00000000000000000000000000000000 P 4055494c55824ee0b04372739fec3560: Log is configured to *not* fsync() on all Append() calls
I20260619 16:44:12.149010 9206 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4055494c55824ee0b04372739fec3560: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 16:44:12.149477 9206 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4055494c55824ee0b04372739fec3560: Bootstrap complete.
I20260619 16:44:12.157891 9206 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4055494c55824ee0b04372739fec3560 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4055494c55824ee0b04372739fec3560" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46703 } }
I20260619 16:44:12.158676 9206 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4055494c55824ee0b04372739fec3560 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4055494c55824ee0b04372739fec3560, State: Initialized, Role: FOLLOWER
I20260619 16:44:12.159235 9206 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4055494c55824ee0b04372739fec3560 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4055494c55824ee0b04372739fec3560" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46703 } }
I20260619 16:44:12.159437 9206 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4055494c55824ee0b04372739fec3560 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:44:12.159536 9206 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4055494c55824ee0b04372739fec3560 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:44:12.159683 9206 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4055494c55824ee0b04372739fec3560 [term 1 FOLLOWER]: Advancing to term 2
I20260619 16:44:12.163199 9206 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4055494c55824ee0b04372739fec3560 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4055494c55824ee0b04372739fec3560" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46703 } }
I20260619 16:44:12.163635 9206 leader_election.cc:302] T 00000000000000000000000000000000 P 4055494c55824ee0b04372739fec3560 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4055494c55824ee0b04372739fec3560; no voters:
I20260619 16:44:12.164199 9206 leader_election.cc:288] T 00000000000000000000000000000000 P 4055494c55824ee0b04372739fec3560 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260619 16:44:12.164384 9210 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4055494c55824ee0b04372739fec3560 [term 2 FOLLOWER]: Leader election won for term 2
I20260619 16:44:12.165279 9210 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4055494c55824ee0b04372739fec3560 [term 2 LEADER]: Becoming Leader. State: Replica: 4055494c55824ee0b04372739fec3560, State: Running, Role: LEADER
I20260619 16:44:12.166033 9206 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4055494c55824ee0b04372739fec3560 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:44:12.165961 9210 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4055494c55824ee0b04372739fec3560 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4055494c55824ee0b04372739fec3560" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46703 } }
I20260619 16:44:12.168619 9212 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4055494c55824ee0b04372739fec3560 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4055494c55824ee0b04372739fec3560. Latest consensus state: current_term: 2 leader_uuid: "4055494c55824ee0b04372739fec3560" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4055494c55824ee0b04372739fec3560" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46703 } } }
I20260619 16:44:12.168977 9212 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4055494c55824ee0b04372739fec3560 [sys.catalog]: This master's current role is: LEADER
I20260619 16:44:12.169426 9211 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4055494c55824ee0b04372739fec3560 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "4055494c55824ee0b04372739fec3560" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4055494c55824ee0b04372739fec3560" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46703 } } }
I20260619 16:44:12.169677 9211 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4055494c55824ee0b04372739fec3560 [sys.catalog]: This master's current role is: LEADER
I20260619 16:44:12.177611 9218 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:44:12.183125 9218 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:44:12.184535 9218 catalog_manager.cc:1259] Loaded cluster ID: b7c38bbacd6d4bb6826f2cf0a69372af
I20260619 16:44:12.184631 9218 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:44:12.188337 9218 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:44:12.189632 9218 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 4055494c55824ee0b04372739fec3560: Loaded TSK: 0
I20260619 16:44:12.191085 9218 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260619 16:44:12.436332 9208 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:44:12.436615 9208 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:44:12.436744 9208 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:44:12.444075 9208 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:44:12.444198 9208 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:44:12.444346 9208 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260619 16:44:12.454002 9208 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34173
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887385903558-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887385903558-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887385903558-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:36879
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887385903558-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=40037
--tserver_master_addrs=127.0.97.62:46703
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887385903558-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:44:12.455518 9208 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:44:12.456775 9208 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:44:12.470872 9236 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:44:12.470847 9234 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:44:12.471030 9233 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:44:12.472441 9208 server_base.cc:1034] running on GCE node
I20260619 16:44:12.473150 9208 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:44:12.474260 9208 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:44:12.475513 9208 hybrid_clock.cc:648] HybridClock initialized: now 1781887452475436 us; error 71 us; skew 500 ppm
I20260619 16:44:12.475862 9208 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:44:12 dist-test-slave-wvz7 krb5kdc[8621](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887452, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:44:12.479372 9208 init.cc:377] Logged in from keytab as oryx/127.0.97.1@KRBTEST.COM (short username oryx)
I20260619 16:44:12.480852 9208 webserver.cc:460] Webserver started at http://127.0.97.1:40037/ using document root <none> and password file <none>
I20260619 16:44:12.481509 9208 fs_manager.cc:359] Metadata directory not provided
I20260619 16:44:12.481631 9208 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:44:12.486943 9208 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 16:44:12.489714 9243 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:44:12.491166 9208 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 16:44:12.491328 9208 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887385903558-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887385903558-388-0/minicluster-data/ts-0/wal
uuid: "42616ba1a2424d8188c6114cd093d7a9"
format_stamp: "Formatted at 2026-06-19 16:44:10 on dist-test-slave-wvz7"
server_key: "c87b9f2d076d34093fc9cbd89f052caf"
server_key_iv: "c4aaca8cdb729788830f124186ae864c"
server_key_version: "encryptionkey@0"
I20260619 16:44:12.492210 9208 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887385903558-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887385903558-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887385903558-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:44:12.505961 9208 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:44:12.509867 9208 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:44:12.510100 9208 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:44:12.511003 9208 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:44:12.512283 9208 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:44:12.512375 9208 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:44:12.512482 9208 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:44:12.512548 9208 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:44:12.540512 9208 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:36879
I20260619 16:44:12.540572 9355 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:36879 every 8 connection(s)
I20260619 16:44:12.541872 9208 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887385903558-388-0/minicluster-data/ts-0/data/info.pb
I20260619 16:44:12.550014 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu as pid 9208
I20260619 16:44:12.550554 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu
/tmp/dist-test-taskEs2EH8/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887385903558-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887385903558-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887385903558-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887385903558-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:39025
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:46703
--webserver_port=34321
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887385903558-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34173
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.97.2
--keytab_file=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887385903558-388-0/krb5kdc/oryx_127.0.97.2.keytab with env {KRB5CCNAME=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887385903558-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887385903558-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887385903558-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 16:44:12 dist-test-slave-wvz7 krb5kdc[8621](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:44:12 dist-test-slave-wvz7 krb5kdc[8621](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 16:44:12.558501 9358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:44:12.544616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46703 (local address 127.0.97.1:35645)
0619 16:44:12.545398 (+ 782us) negotiation.cc:107] Waiting for socket to connect
0619 16:44:12.545449 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0619 16:44:12.547034 (+ 1585us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:44:12.548313 (+ 1279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:44:12.548322 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:44:12.548870 (+ 548us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:44:12.549370 (+ 500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:44:12.549386 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:44:12.551053 (+ 1667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:44:12.551056 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:44:12.552238 (+ 1182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:44:12.552250 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:44:12.553214 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:44:12.553221 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:44:12.553462 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:44:12.554067 (+ 605us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:44:12.554092 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:44:12.557435 (+ 3343us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:46703: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":492,"thread_start_us":312,"threads_started":1}
W20260619 16:44:12.558987 9359 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:44:12.544565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:35645 (local address 127.0.97.62:46703)
0619 16:44:12.545394 (+ 829us) server_negotiation.cc:206] Beginning negotiation
0619 16:44:12.545409 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 16:44:12.545561 (+ 152us) server_negotiation.cc:407] Connection header received
0619 16:44:12.547540 (+ 1979us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:44:12.547553 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:44:12.547855 (+ 302us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:44:12.548154 (+ 299us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:44:12.549562 (+ 1408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:44:12.550910 (+ 1348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:44:12.552385 (+ 1475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:44:12.553005 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:44:12.553168 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:44:12.557880 (+ 4712us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:35645: BlockingRecv error: recv got EOF from 127.0.97.1:35645 (error 108)
Metrics: {"server-negotiator.queue_time_us":594,"thread_start_us":369,"threads_started":1}
W20260619 16:44:12.559290 9356 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:46703 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.97.62:46703: Client connection negotiation failed: client connection to 127.0.97.62:46703: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Jun 19 16:44:12 dist-test-slave-wvz7 krb5kdc[8621](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:44:12 dist-test-slave-wvz7 krb5kdc[8621](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 16:44:12.567662 9358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:44:12.559838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46703 (local address 127.0.97.1:45547)
0619 16:44:12.559994 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0619 16:44:12.560013 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 16:44:12.560102 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:44:12.560353 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:44:12.560356 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:44:12.560623 (+ 267us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:44:12.560786 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:44:12.560793 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:44:12.562027 (+ 1234us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:44:12.562031 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:44:12.562924 (+ 893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:44:12.562935 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:44:12.563766 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:44:12.563770 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:44:12.563889 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:44:12.564399 (+ 510us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:44:12.564421 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:44:12.567456 (+ 3035us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:46703: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":48}
W20260619 16:44:12.568251 9359 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:44:12.559956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:45547 (local address 127.0.97.62:46703)
0619 16:44:12.560095 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0619 16:44:12.560099 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:44:12.560118 (+ 19us) server_negotiation.cc:407] Connection header received
0619 16:44:12.560184 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:44:12.560188 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:44:12.560251 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:44:12.560337 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:44:12.560901 (+ 564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:44:12.561901 (+ 1000us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:44:12.563055 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:44:12.563639 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:44:12.563748 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:44:12.567662 (+ 3914us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:45547: BlockingRecv error: recv got EOF from 127.0.97.1:45547 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
Jun 19 16:44:12 dist-test-slave-wvz7 krb5kdc[8621](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:44:12 dist-test-slave-wvz7 krb5kdc[8621](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 16:44:12.575862 9358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:44:12.568344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46703 (local address 127.0.97.1:47287)
0619 16:44:12.568575 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0619 16:44:12.568613 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0619 16:44:12.568733 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:44:12.569025 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:44:12.569030 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:44:12.569311 (+ 281us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:44:12.569448 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:44:12.569456 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:44:12.570462 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:44:12.570466 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:44:12.571376 (+ 910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:44:12.571386 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:44:12.572038 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:44:12.572044 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:44:12.572200 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:44:12.572728 (+ 528us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:44:12.572745 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:44:12.575571 (+ 2826us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:46703: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":126}
W20260619 16:44:12.576050 9359 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:44:12.568412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:47287 (local address 127.0.97.62:46703)
0619 16:44:12.568540 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0619 16:44:12.568546 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:44:12.568740 (+ 194us) server_negotiation.cc:407] Connection header received
0619 16:44:12.568846 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:44:12.568852 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:44:12.568909 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:44:12.568992 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:44:12.569581 (+ 589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:44:12.570321 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:44:12.571507 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:44:12.571899 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:44:12.572004 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:44:12.575800 (+ 3796us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:47287: BlockingRecv error: recv got EOF from 127.0.97.1:47287 (error 108)
Metrics: {"server-negotiator.queue_time_us":31}
W20260619 16:44:12.576303 9356 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260619 16:44:12.851742 9360 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:44:12.852010 9360 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:44:12.852135 9360 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:44:12.859023 9360 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:44:12.859133 9360 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:44:12.859280 9360 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260619 16:44:12.868680 9360 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34173
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887385903558-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887385903558-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887385903558-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:39025
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887385903558-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=34321
--tserver_master_addrs=127.0.97.62:46703
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887385903558-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:44:12.869951 9360 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:44:12.871187 9360 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:44:12.883016 9365 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:44:12.883162 9366 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:44:12.883139 9368 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:44:12.883778 9360 server_base.cc:1034] running on GCE node
I20260619 16:44:12.884286 9360 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:44:12.885121 9360 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:44:12.886303 9360 hybrid_clock.cc:648] HybridClock initialized: now 1781887452886255 us; error 35 us; skew 500 ppm
I20260619 16:44:12.886566 9360 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:44:12 dist-test-slave-wvz7 krb5kdc[8621](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887452, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:44:12.889286 9360 init.cc:377] Logged in from keytab as oryx/127.0.97.2@KRBTEST.COM (short username oryx)
I20260619 16:44:12.890558 9360 webserver.cc:460] Webserver started at http://127.0.97.2:34321/ using document root <none> and password file <none>
I20260619 16:44:12.891316 9360 fs_manager.cc:359] Metadata directory not provided
I20260619 16:44:12.891456 9360 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:44:12.898294 9360 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260619 16:44:12.900960 9375 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:44:12.902379 9360 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260619 16:44:12.902527 9360 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887385903558-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887385903558-388-0/minicluster-data/ts-1/wal
uuid: "df3833d59e3244a3b0ae9f68ee2d9c94"
format_stamp: "Formatted at 2026-06-19 16:44:10 on dist-test-slave-wvz7"
server_key: "9343e96794e99a2e89a08b898c14f75d"
server_key_iv: "3221070840d7a7cb1f314bef16c65951"
server_key_version: "encryptionkey@0"
I20260619 16:44:12.903409 9360 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887385903558-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887385903558-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887385903558-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:44:12.927956 9360 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:44:12.931540 9360 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:44:12.931790 9360 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:44:12.932674 9360 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:44:12.933957 9360 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:44:12.934054 9360 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:44:12.934150 9360 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:44:12.934219 9360 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:44:12.961956 9360 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:39025
I20260619 16:44:12.962009 9487 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:39025 every 8 connection(s)
I20260619 16:44:12.963371 9360 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887385903558-388-0/minicluster-data/ts-1/data/info.pb
I20260619 16:44:12.970980 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu as pid 9360
I20260619 16:44:12.971522 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu
/tmp/dist-test-taskEs2EH8/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887385903558-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887385903558-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887385903558-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887385903558-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:45155
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:46703
--webserver_port=40441
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887385903558-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34173
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.97.3
--keytab_file=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887385903558-388-0/krb5kdc/oryx_127.0.97.3.keytab with env {KRB5CCNAME=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887385903558-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887385903558-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887385903558-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 16:44:12 dist-test-slave-wvz7 krb5kdc[8621](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:44:12 dist-test-slave-wvz7 krb5kdc[8621](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
W20260619 16:44:12.983430 9359 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:44:12.965941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:59071 (local address 127.0.97.62:46703)
0619 16:44:12.966109 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0619 16:44:12.966116 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:44:12.967413 (+ 1297us) server_negotiation.cc:407] Connection header received
0619 16:44:12.969161 (+ 1748us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:44:12.969166 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:44:12.969240 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:44:12.969332 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:44:12.970893 (+ 1561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:44:12.971671 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:44:12.973822 (+ 2151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:44:12.974223 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:44:12.974336 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:44:12.983187 (+ 8851us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:59071: BlockingRecv error: recv got EOF from 127.0.97.2:59071 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
I20260619 16:44:12.984138 9490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:44:12.966520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46703 (local address 127.0.97.2:59071)
0619 16:44:12.967260 (+ 740us) negotiation.cc:107] Waiting for socket to connect
0619 16:44:12.967305 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0619 16:44:12.968918 (+ 1613us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:44:12.969561 (+ 643us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:44:12.969570 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:44:12.970116 (+ 546us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:44:12.970703 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:44:12.970727 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:44:12.971853 (+ 1126us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:44:12.971857 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:44:12.973657 (+ 1800us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:44:12.973670 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:44:12.974424 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:44:12.974430 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:44:12.974676 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:44:12.979515 (+ 4839us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:44:12.979538 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:44:12.983001 (+ 3463us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:46703: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":448,"thread_start_us":264,"threads_started":1}
W20260619 16:44:12.984782 9488 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:46703 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.97.62:46703: Client connection negotiation failed: client connection to 127.0.97.62:46703: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Jun 19 16:44:12 dist-test-slave-wvz7 krb5kdc[8621](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:44:12 dist-test-slave-wvz7 krb5kdc[8621](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 16:44:12.992457 9490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:44:12.985412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46703 (local address 127.0.97.2:38939)
0619 16:44:12.985573 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0619 16:44:12.985592 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 16:44:12.985690 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:44:12.985990 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:44:12.985995 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:44:12.986292 (+ 297us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:44:12.986422 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:44:12.986430 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:44:12.987349 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:44:12.987353 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:44:12.988208 (+ 855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:44:12.988219 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:44:12.988886 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:44:12.988891 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:44:12.989033 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:44:12.989573 (+ 540us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:44:12.989588 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:44:12.992204 (+ 2616us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:46703: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":51}
W20260619 16:44:12.992627 9359 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:44:12.985471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:38939 (local address 127.0.97.62:46703)
0619 16:44:12.985612 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0619 16:44:12.985616 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:44:12.985643 (+ 27us) server_negotiation.cc:407] Connection header received
0619 16:44:12.985808 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:44:12.985814 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:44:12.985877 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:44:12.985958 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:44:12.986548 (+ 590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:44:12.987222 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:44:12.988337 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:44:12.988733 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:44:12.988843 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:44:12.992426 (+ 3583us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:38939: BlockingRecv error: recv got EOF from 127.0.97.2:38939 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Jun 19 16:44:12 dist-test-slave-wvz7 krb5kdc[8621](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:44:12 dist-test-slave-wvz7 krb5kdc[8621](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 16:44:13.001729 9490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:44:12.993319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46703 (local address 127.0.97.2:46999)
0619 16:44:12.993477 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0619 16:44:12.993496 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 16:44:12.993588 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:44:12.993886 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:44:12.993891 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:44:12.994152 (+ 261us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:44:12.994282 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:44:12.994290 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:44:12.995294 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:44:12.995301 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:44:12.996949 (+ 1648us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:44:12.996966 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:44:12.997620 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:44:12.997626 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:44:12.997855 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:44:12.998416 (+ 561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:44:12.998441 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:44:13.001452 (+ 3011us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:46703: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":48}
W20260619 16:44:13.001879 9359 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:44:12.993389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:46999 (local address 127.0.97.62:46703)
0619 16:44:12.993536 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0619 16:44:12.993541 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:44:12.993557 (+ 16us) server_negotiation.cc:407] Connection header received
0619 16:44:12.993685 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:44:12.993690 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:44:12.993773 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:44:12.993853 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:44:12.994411 (+ 558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:44:12.995091 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:44:12.997086 (+ 1995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:44:12.997485 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:44:12.997574 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:44:13.001662 (+ 4088us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:46999: BlockingRecv error: recv got EOF from 127.0.97.2:46999 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
W20260619 16:44:13.002327 9488 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260619 16:44:13.295253 9491 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:44:13.295512 9491 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:44:13.295607 9491 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:44:13.302583 9491 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:44:13.302685 9491 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:44:13.302800 9491 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260619 16:44:13.312037 9491 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34173
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887385903558-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887385903558-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887385903558-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:45155
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887385903558-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=40441
--tserver_master_addrs=127.0.97.62:46703
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887385903558-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:44:13.313187 9491 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:44:13.314354 9491 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:44:13.325919 9499 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:44:13.326222 9496 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:44:13.326260 9497 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:44:13.326947 9491 server_base.cc:1034] running on GCE node
I20260619 16:44:13.327524 9491 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:44:13.328406 9491 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:44:13.329595 9491 hybrid_clock.cc:648] HybridClock initialized: now 1781887453329552 us; error 39 us; skew 500 ppm
I20260619 16:44:13.330011 9491 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:44:13 dist-test-slave-wvz7 krb5kdc[8621](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887453, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:44:13.332880 9491 init.cc:377] Logged in from keytab as oryx/127.0.97.3@KRBTEST.COM (short username oryx)
I20260619 16:44:13.334246 9491 webserver.cc:460] Webserver started at http://127.0.97.3:40441/ using document root <none> and password file <none>
I20260619 16:44:13.334873 9491 fs_manager.cc:359] Metadata directory not provided
I20260619 16:44:13.334983 9491 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:44:13.339876 9491 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.002s
I20260619 16:44:13.342260 9506 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:44:13.343469 9491 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 16:44:13.343595 9491 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887385903558-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887385903558-388-0/minicluster-data/ts-2/wal
uuid: "dfdff7e441fc4bdda1489ff74e8bfa74"
format_stamp: "Formatted at 2026-06-19 16:44:11 on dist-test-slave-wvz7"
server_key: "ebc37e54066dc04a52b0592106f10911"
server_key_iv: "f4986a8592bd1e36ff365e5136be9b38"
server_key_version: "encryptionkey@0"
I20260619 16:44:13.344465 9491 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887385903558-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887385903558-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887385903558-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:44:13.379367 9491 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:44:13.383179 9491 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:44:13.383387 9491 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:44:13.384271 9491 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:44:13.385541 9491 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:44:13.385623 9491 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:44:13.385699 9491 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:44:13.385797 9491 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260619 16:44:13.416666 9491 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:45155
I20260619 16:44:13.416707 9618 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:45155 every 8 connection(s)
I20260619 16:44:13.418167 9491 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887385903558-388-0/minicluster-data/ts-2/data/info.pb
I20260619 16:44:13.422245 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu as pid 9491
Jun 19 16:44:13 dist-test-slave-wvz7 krb5kdc[8621](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:44:13 dist-test-slave-wvz7 krb5kdc[8621](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
W20260619 16:44:13.435475 9359 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:44:13.420437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:34735 (local address 127.0.97.62:46703)
0619 16:44:13.420614 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0619 16:44:13.420618 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:44:13.421784 (+ 1166us) server_negotiation.cc:407] Connection header received
0619 16:44:13.424038 (+ 2254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:44:13.424042 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:44:13.424091 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:44:13.424158 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:44:13.426137 (+ 1979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:44:13.427425 (+ 1288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:44:13.429563 (+ 2138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:44:13.430157 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:44:13.430253 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:44:13.435199 (+ 4946us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:34735: BlockingRecv error: recv got EOF from 127.0.97.3:34735 (error 108)
Metrics: {"mutex_wait_us":105,"server-negotiator.queue_time_us":66,"spinlock_wait_cycles":2816}
I20260619 16:44:13.435940 9621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:44:13.420897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46703 (local address 127.0.97.3:34735)
0619 16:44:13.421602 (+ 705us) negotiation.cc:107] Waiting for socket to connect
0619 16:44:13.421668 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0619 16:44:13.423812 (+ 2144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:44:13.424453 (+ 641us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:44:13.424465 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:44:13.425180 (+ 715us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:44:13.425998 (+ 818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:44:13.426023 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:44:13.427609 (+ 1586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:44:13.427614 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:44:13.429447 (+ 1833us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:44:13.429458 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:44:13.430375 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:44:13.430384 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:44:13.430607 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:44:13.431148 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:44:13.431177 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:44:13.434974 (+ 3797us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:46703: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":462,"thread_start_us":270,"threads_started":1}
W20260619 16:44:13.436556 9619 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:46703 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.97.62:46703: Client connection negotiation failed: client connection to 127.0.97.62:46703: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
I20260619 16:44:13.439431 9622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:44:13.423095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43274 (local address 127.0.97.62:46703)
0619 16:44:13.423511 (+ 416us) server_negotiation.cc:206] Beginning negotiation
0619 16:44:13.423517 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:44:13.423797 (+ 280us) server_negotiation.cc:407] Connection header received
0619 16:44:13.423972 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:44:13.423976 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:44:13.424049 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:44:13.424151 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:44:13.426136 (+ 1985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:44:13.427425 (+ 1289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:44:13.429209 (+ 1784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:44:13.429667 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:44:13.429804 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:44:13.432966 (+ 3162us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:44:13.432974 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:44:13.432989 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:44:13.433025 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:44:13.435977 (+ 2952us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:44:13.436867 (+ 890us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:44:13.436874 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:44:13.436880 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:44:13.437046 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:44:13.437568 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:44:13.437573 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:44:13.437576 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:44:13.438051 (+ 475us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:44:13.438369 (+ 318us) server_negotiation.cc:1035] Waiting for connection context
0619 16:44:13.438714 (+ 345us) server_negotiation.cc:299] Negotiation successful
0619 16:44:13.439058 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":30,"server-negotiator.queue_time_us":317,"thread_start_us":194,"threads_started":1}
Jun 19 16:44:13 dist-test-slave-wvz7 krb5kdc[8621](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:44:13 dist-test-slave-wvz7 krb5kdc[8621](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 16:44:13.444226 9621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:44:13.437057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46703 (local address 127.0.97.3:44317)
0619 16:44:13.437197 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0619 16:44:13.437214 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 16:44:13.437290 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:44:13.437611 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:44:13.437615 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:44:13.437973 (+ 358us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:44:13.438096 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:44:13.438103 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:44:13.439108 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:44:13.439114 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:44:13.440138 (+ 1024us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:44:13.440150 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:44:13.440915 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:44:13.440920 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:44:13.441084 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:44:13.441571 (+ 487us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:44:13.441592 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:44:13.444050 (+ 2458us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:46703: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":41}
W20260619 16:44:13.444476 9359 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:44:13.437249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:44317 (local address 127.0.97.62:46703)
0619 16:44:13.437403 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0619 16:44:13.437408 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:44:13.437421 (+ 13us) server_negotiation.cc:407] Connection header received
0619 16:44:13.437454 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:44:13.437457 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:44:13.437503 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:44:13.437574 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:44:13.438217 (+ 643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:44:13.438970 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:44:13.440274 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:44:13.440756 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:44:13.440843 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:44:13.444252 (+ 3409us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:44317: BlockingRecv error: recv got EOF from 127.0.97.3:44317 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Jun 19 16:44:13 dist-test-slave-wvz7 krb5kdc[8621](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:44:13 dist-test-slave-wvz7 krb5kdc[8621](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 16:44:13.451610 9621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:44:13.444883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46703 (local address 127.0.97.3:45131)
0619 16:44:13.445056 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0619 16:44:13.445076 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 16:44:13.445168 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:44:13.445447 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:44:13.445451 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:44:13.445701 (+ 250us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:44:13.445904 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:44:13.445913 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:44:13.446821 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:44:13.446825 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:44:13.447790 (+ 965us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:44:13.447801 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:44:13.448454 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:44:13.448459 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:44:13.448571 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:44:13.449036 (+ 465us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:44:13.449051 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:44:13.451416 (+ 2365us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:46703: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":58}
W20260619 16:44:13.451854 9359 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:44:13.444966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:45131 (local address 127.0.97.62:46703)
0619 16:44:13.445170 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0619 16:44:13.445173 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 16:44:13.445188 (+ 15us) server_negotiation.cc:407] Connection header received
0619 16:44:13.445257 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:44:13.445261 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:44:13.445306 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:44:13.445376 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:44:13.446026 (+ 650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:44:13.446682 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:44:13.447900 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:44:13.448344 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:44:13.448448 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:44:13.451591 (+ 3143us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:45131: BlockingRecv error: recv got EOF from 127.0.97.3:45131 (error 108)
Metrics: {"server-negotiator.queue_time_us":112}
W20260619 16:44:13.451869 9619 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Jun 19 16:44:13 dist-test-slave-wvz7 krb5kdc[8621](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:44:13 dist-test-slave-wvz7 krb5kdc[8621](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 16:44:13.585191 9624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:44:13.576925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46703 (local address 127.0.97.1:35047)
0619 16:44:13.577476 (+ 551us) negotiation.cc:107] Waiting for socket to connect
0619 16:44:13.577497 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 16:44:13.577625 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:44:13.578028 (+ 403us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:44:13.578033 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:44:13.578496 (+ 463us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:44:13.578678 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:44:13.578688 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:44:13.579659 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:44:13.579664 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:44:13.580576 (+ 912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:44:13.580588 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:44:13.581295 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:44:13.581301 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:44:13.581442 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:44:13.581973 (+ 531us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:44:13.581990 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:44:13.584896 (+ 2906us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:46703: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":421,"thread_start_us":291,"threads_started":1}
W20260619 16:44:13.585305 9359 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:44:13.577006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:35047 (local address 127.0.97.62:46703)
0619 16:44:13.577153 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0619 16:44:13.577157 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:44:13.577565 (+ 408us) server_negotiation.cc:407] Connection header received
0619 16:44:13.577796 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:44:13.577802 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:44:13.577874 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:44:13.577976 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:44:13.578841 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:44:13.579529 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:44:13.580704 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:44:13.581133 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:44:13.581246 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:44:13.585089 (+ 3843us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:35047: BlockingRecv error: recv got EOF from 127.0.97.1:35047 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Jun 19 16:44:14 dist-test-slave-wvz7 krb5kdc[8621](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:44:14 dist-test-slave-wvz7 krb5kdc[8621](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 16:44:14.011092 9625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:44:14.002889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46703 (local address 127.0.97.2:42759)
0619 16:44:14.003451 (+ 562us) negotiation.cc:107] Waiting for socket to connect
0619 16:44:14.003467 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 16:44:14.003578 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:44:14.003855 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:44:14.003858 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:44:14.004269 (+ 411us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:44:14.004452 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:44:14.004460 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:44:14.005468 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:44:14.005472 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:44:14.006483 (+ 1011us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:44:14.006502 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:44:14.007246 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:44:14.007252 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:44:14.007411 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:44:14.008009 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:44:14.008027 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:44:14.010878 (+ 2851us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:46703: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":444,"thread_start_us":318,"threads_started":1}
W20260619 16:44:14.011276 9359 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:44:14.002980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:42759 (local address 127.0.97.62:46703)
0619 16:44:14.003213 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0619 16:44:14.003217 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:44:14.003542 (+ 325us) server_negotiation.cc:407] Connection header received
0619 16:44:14.003666 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:44:14.003669 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:44:14.003728 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:44:14.003816 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:44:14.004605 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:44:14.005328 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:44:14.006634 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:44:14.007076 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:44:14.007190 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:44:14.011069 (+ 3879us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:42759: BlockingRecv error: recv got EOF from 127.0.97.2:42759 (error 108)
Metrics: {"server-negotiator.queue_time_us":86}
Jun 19 16:44:14 dist-test-slave-wvz7 krb5kdc[8621](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:44:14 dist-test-slave-wvz7 krb5kdc[8621](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 16:44:14.460228 9626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:44:14.452433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46703 (local address 127.0.97.3:49659)
0619 16:44:14.452954 (+ 521us) negotiation.cc:107] Waiting for socket to connect
0619 16:44:14.452971 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 16:44:14.453095 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:44:14.453395 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:44:14.453399 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:44:14.453938 (+ 539us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:44:14.454141 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:44:14.454154 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:44:14.455124 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:44:14.455128 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:44:14.456115 (+ 987us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:44:14.456127 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:44:14.456803 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:44:14.456808 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:44:14.456932 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:44:14.457474 (+ 542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:44:14.457492 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:44:14.460009 (+ 2517us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:46703: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":408,"thread_start_us":285,"threads_started":1}
W20260619 16:44:14.460569 9359 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:44:14.452513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:49659 (local address 127.0.97.62:46703)
0619 16:44:14.452706 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0619 16:44:14.452710 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:44:14.453054 (+ 344us) server_negotiation.cc:407] Connection header received
0619 16:44:14.453210 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:44:14.453214 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:44:14.453272 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:44:14.453357 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:44:14.454272 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:44:14.454987 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:44:14.456246 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:44:14.456622 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:44:14.456716 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:44:14.460280 (+ 3564us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:49659: BlockingRecv error: recv got EOF from 127.0.97.3:49659 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
Jun 19 16:44:14 dist-test-slave-wvz7 krb5kdc[8621](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:44:14 dist-test-slave-wvz7 krb5kdc[8621](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 16:44:14.594621 9627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:44:14.586288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46703 (local address 127.0.97.1:57459)
0619 16:44:14.586754 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0619 16:44:14.586770 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 16:44:14.586896 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:44:14.587232 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:44:14.587235 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:44:14.587590 (+ 355us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:44:14.587733 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:44:14.587740 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:44:14.588734 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:44:14.588738 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:44:14.589635 (+ 897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:44:14.589646 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:44:14.590354 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:44:14.590359 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:44:14.590470 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:44:14.591119 (+ 649us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:44:14.591138 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:44:14.594234 (+ 3096us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:46703: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":366,"thread_start_us":257,"threads_started":1}
W20260619 16:44:14.594725 9359 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:44:14.586378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:57459 (local address 127.0.97.62:46703)
0619 16:44:14.586597 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0619 16:44:14.586603 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:44:14.586873 (+ 270us) server_negotiation.cc:407] Connection header received
0619 16:44:14.587054 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:44:14.587058 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:44:14.587120 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:44:14.587201 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:44:14.587866 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:44:14.588613 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:44:14.589798 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:44:14.590205 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:44:14.590311 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:44:14.594480 (+ 4169us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:57459: BlockingRecv error: recv got EOF from 127.0.97.1:57459 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
Jun 19 16:44:15 dist-test-slave-wvz7 krb5kdc[8621](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:44:15 dist-test-slave-wvz7 krb5kdc[8621](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 16:44:15.020539 9628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:44:15.011984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46703 (local address 127.0.97.2:44935)
0619 16:44:15.012466 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0619 16:44:15.012482 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 16:44:15.012596 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:44:15.012961 (+ 365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:44:15.012966 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:44:15.013430 (+ 464us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:44:15.013582 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:44:15.013591 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:44:15.014800 (+ 1209us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:44:15.014827 (+ 27us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:44:15.016069 (+ 1242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:44:15.016081 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:44:15.016990 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:44:15.016995 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:44:15.017116 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:44:15.017637 (+ 521us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:44:15.017653 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:44:15.020303 (+ 2650us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:46703: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":370,"thread_start_us":254,"threads_started":1}
W20260619 16:44:15.020768 9359 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:44:15.012113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:44935 (local address 127.0.97.62:46703)
0619 16:44:15.012274 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0619 16:44:15.012279 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:44:15.012550 (+ 271us) server_negotiation.cc:407] Connection header received
0619 16:44:15.012682 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:44:15.012686 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:44:15.012773 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:44:15.012869 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:44:15.013785 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:44:15.014657 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:44:15.016202 (+ 1545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:44:15.016849 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:44:15.016942 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:44:15.020522 (+ 3580us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:44935: BlockingRecv error: recv got EOF from 127.0.97.2:44935 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Jun 19 16:44:15 dist-test-slave-wvz7 krb5kdc[8621](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:44:15 dist-test-slave-wvz7 krb5kdc[8621](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 16:44:15.469826 9629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:44:15.461493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46703 (local address 127.0.97.3:56383)
0619 16:44:15.462071 (+ 578us) negotiation.cc:107] Waiting for socket to connect
0619 16:44:15.462090 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 16:44:15.462219 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:44:15.462537 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:44:15.462542 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:44:15.462910 (+ 368us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:44:15.463051 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:44:15.463059 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:44:15.464066 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:44:15.464070 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:44:15.464951 (+ 881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:44:15.464961 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:44:15.465613 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:44:15.465617 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:44:15.465724 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:44:15.466359 (+ 635us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:44:15.466383 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:44:15.469480 (+ 3097us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:46703: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":461,"thread_start_us":334,"threads_started":1}
W20260619 16:44:15.470011 9359 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:44:15.461681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:56383 (local address 127.0.97.62:46703)
0619 16:44:15.461903 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0619 16:44:15.461909 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:44:15.462199 (+ 290us) server_negotiation.cc:407] Connection header received
0619 16:44:15.462308 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:44:15.462313 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:44:15.462391 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:44:15.462492 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:44:15.463183 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:44:15.463949 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:44:15.465080 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:44:15.465496 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:44:15.465601 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:44:15.469715 (+ 4114us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:56383: BlockingRecv error: recv got EOF from 127.0.97.3:56383 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Jun 19 16:44:15 dist-test-slave-wvz7 krb5kdc[8621](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:44:15 dist-test-slave-wvz7 krb5kdc[8621](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 16:44:15.603356 9630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:44:15.595595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46703 (local address 127.0.97.1:53969)
0619 16:44:15.596037 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0619 16:44:15.596054 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 16:44:15.596183 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:44:15.596501 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:44:15.596505 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:44:15.596908 (+ 403us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:44:15.597079 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:44:15.597088 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:44:15.598092 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:44:15.598096 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:44:15.599002 (+ 906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:44:15.599013 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:44:15.599625 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:44:15.599629 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:44:15.599777 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:44:15.600289 (+ 512us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:44:15.600305 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:44:15.603032 (+ 2727us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:46703: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":345,"thread_start_us":235,"threads_started":1}
W20260619 16:44:15.603471 9359 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:44:15.595712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:53969 (local address 127.0.97.62:46703)
0619 16:44:15.595909 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0619 16:44:15.595916 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:44:15.596131 (+ 215us) server_negotiation.cc:407] Connection header received
0619 16:44:15.596276 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:44:15.596280 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:44:15.596345 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:44:15.596420 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:44:15.597217 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:44:15.597971 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:44:15.599115 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:44:15.599507 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:44:15.599594 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:44:15.603227 (+ 3633us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:53969: BlockingRecv error: recv got EOF from 127.0.97.1:53969 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
Jun 19 16:44:16 dist-test-slave-wvz7 krb5kdc[8621](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:44:16 dist-test-slave-wvz7 krb5kdc[8621](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 16:44:16.029407 9631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:44:16.021418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46703 (local address 127.0.97.2:35029)
0619 16:44:16.021962 (+ 544us) negotiation.cc:107] Waiting for socket to connect
0619 16:44:16.021978 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 16:44:16.022088 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:44:16.022359 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:44:16.022362 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:44:16.022674 (+ 312us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:44:16.022856 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:44:16.022865 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:44:16.023898 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:44:16.023902 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:44:16.024768 (+ 866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:44:16.024778 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:44:16.025456 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:44:16.025460 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:44:16.025586 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:44:16.026202 (+ 616us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:44:16.026219 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:44:16.029156 (+ 2937us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:46703: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":438,"thread_start_us":319,"threads_started":1}
W20260619 16:44:16.029543 9359 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:44:16.021523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:35029 (local address 127.0.97.62:46703)
0619 16:44:16.021758 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0619 16:44:16.021764 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:44:16.022074 (+ 310us) server_negotiation.cc:407] Connection header received
0619 16:44:16.022178 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:44:16.022183 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:44:16.022246 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:44:16.022327 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:44:16.022999 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:44:16.023759 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:44:16.024907 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:44:16.025322 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:44:16.025415 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:44:16.029350 (+ 3935us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:35029: BlockingRecv error: recv got EOF from 127.0.97.2:35029 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
Jun 19 16:44:16 dist-test-slave-wvz7 krb5kdc[8621](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:44:16 dist-test-slave-wvz7 krb5kdc[8621](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 16:44:16.479511 9632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:44:16.471045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46703 (local address 127.0.97.3:43227)
0619 16:44:16.471548 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0619 16:44:16.471566 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 16:44:16.471702 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:44:16.472128 (+ 426us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:44:16.472132 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:44:16.472562 (+ 430us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:44:16.472741 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:44:16.472750 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:44:16.473918 (+ 1168us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:44:16.473924 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:44:16.474845 (+ 921us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:44:16.474856 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:44:16.475561 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:44:16.475567 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:44:16.475730 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:44:16.476294 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:44:16.476311 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:44:16.479218 (+ 2907us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:46703: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":394,"thread_start_us":275,"threads_started":1}
W20260619 16:44:16.479602 9359 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:44:16.471144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:43227 (local address 127.0.97.62:46703)
0619 16:44:16.471394 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0619 16:44:16.471422 (+ 28us) server_negotiation.cc:399] Waiting for connection header
0619 16:44:16.471635 (+ 213us) server_negotiation.cc:407] Connection header received
0619 16:44:16.471881 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:44:16.471887 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:44:16.471991 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:44:16.472092 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:44:16.472878 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:44:16.473802 (+ 924us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:44:16.475008 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:44:16.475434 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:44:16.475538 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:44:16.479422 (+ 3884us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:43227: BlockingRecv error: recv got EOF from 127.0.97.3:43227 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
Jun 19 16:44:16 dist-test-slave-wvz7 krb5kdc[8621](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:44:16 dist-test-slave-wvz7 krb5kdc[8621](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 16:44:16.611928 9633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:44:16.604150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46703 (local address 127.0.97.1:46325)
0619 16:44:16.604620 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0619 16:44:16.604634 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 16:44:16.604796 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:44:16.605095 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:44:16.605100 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:44:16.605464 (+ 364us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:44:16.605603 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:44:16.605611 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:44:16.606584 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:44:16.606588 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:44:16.607593 (+ 1005us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:44:16.607605 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:44:16.608301 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:44:16.608307 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:44:16.608511 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:44:16.609070 (+ 559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:44:16.609087 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:44:16.611697 (+ 2610us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:46703: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":380,"thread_start_us":270,"threads_started":1}
W20260619 16:44:16.612169 9359 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:44:16.604246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:46325 (local address 127.0.97.62:46703)
0619 16:44:16.604440 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0619 16:44:16.604448 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 16:44:16.604718 (+ 270us) server_negotiation.cc:407] Connection header received
0619 16:44:16.604912 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:44:16.604916 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:44:16.604974 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:44:16.605057 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:44:16.605764 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:44:16.606462 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:44:16.607739 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:44:16.608163 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:44:16.608285 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:44:16.611879 (+ 3594us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:46325: BlockingRecv error: recv got EOF from 127.0.97.1:46325 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
Jun 19 16:44:17 dist-test-slave-wvz7 krb5kdc[8621](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:44:17 dist-test-slave-wvz7 krb5kdc[8621](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 16:44:17.037891 9634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:44:17.030314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46703 (local address 127.0.97.2:48649)
0619 16:44:17.030725 (+ 411us) negotiation.cc:107] Waiting for socket to connect
0619 16:44:17.030741 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 16:44:17.030861 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:44:17.031157 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:44:17.031161 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:44:17.031506 (+ 345us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:44:17.031653 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:44:17.031661 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:44:17.032608 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:44:17.032612 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:44:17.033583 (+ 971us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:44:17.033594 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:44:17.034344 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:44:17.034348 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:44:17.034495 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:44:17.035089 (+ 594us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:44:17.035105 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:44:17.037648 (+ 2543us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:46703: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":309,"thread_start_us":207,"threads_started":1}
W20260619 16:44:17.038081 9359 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:44:17.030376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:48649 (local address 127.0.97.62:46703)
0619 16:44:17.030572 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0619 16:44:17.030578 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:44:17.030819 (+ 241us) server_negotiation.cc:407] Connection header received
0619 16:44:17.030947 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:44:17.030950 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:44:17.031008 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:44:17.031084 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:44:17.031780 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:44:17.032491 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:44:17.033779 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:44:17.034194 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:44:17.034294 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:44:17.037875 (+ 3581us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:48649: BlockingRecv error: recv got EOF from 127.0.97.2:48649 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Jun 19 16:44:17 dist-test-slave-wvz7 krb5kdc[8621](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:44:17 dist-test-slave-wvz7 krb5kdc[8621](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 16:44:17.489101 9635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:44:17.480511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46703 (local address 127.0.97.3:37429)
0619 16:44:17.481005 (+ 494us) negotiation.cc:107] Waiting for socket to connect
0619 16:44:17.481024 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 16:44:17.481144 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:44:17.481481 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:44:17.481485 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:44:17.481969 (+ 484us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:44:17.482147 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:44:17.482156 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:44:17.483270 (+ 1114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:44:17.483274 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:44:17.484221 (+ 947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:44:17.484233 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:44:17.485136 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:44:17.485141 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:44:17.485328 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:44:17.486042 (+ 714us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:44:17.486064 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:44:17.488785 (+ 2721us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:46703: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":374,"thread_start_us":266,"threads_started":1}
W20260619 16:44:17.489221 9359 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:44:17.480560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:37429 (local address 127.0.97.62:46703)
0619 16:44:17.480819 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0619 16:44:17.480826 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:44:17.481102 (+ 276us) server_negotiation.cc:407] Connection header received
0619 16:44:17.481235 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:44:17.481244 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:44:17.481317 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:44:17.481402 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:44:17.482311 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:44:17.483127 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:44:17.484378 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:44:17.484968 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:44:17.485086 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:44:17.488992 (+ 3906us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:37429: BlockingRecv error: recv got EOF from 127.0.97.3:37429 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
Jun 19 16:44:17 dist-test-slave-wvz7 krb5kdc[8621](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:44:17 dist-test-slave-wvz7 krb5kdc[8621](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 16:44:17.620632 9636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:44:17.612855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46703 (local address 127.0.97.1:44603)
0619 16:44:17.613380 (+ 525us) negotiation.cc:107] Waiting for socket to connect
0619 16:44:17.613396 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 16:44:17.613510 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:44:17.613790 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:44:17.613793 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:44:17.614186 (+ 393us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:44:17.614344 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:44:17.614352 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:44:17.615403 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:44:17.615407 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:44:17.616301 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:44:17.616313 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:44:17.617025 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:44:17.617030 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:44:17.617158 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:44:17.617697 (+ 539us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:44:17.617719 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:44:17.620403 (+ 2684us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:46703: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":390,"thread_start_us":239,"threads_started":1}
W20260619 16:44:17.620728 9359 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:44:17.612924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:44603 (local address 127.0.97.62:46703)
0619 16:44:17.613157 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0619 16:44:17.613163 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:44:17.613498 (+ 335us) server_negotiation.cc:407] Connection header received
0619 16:44:17.613605 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:44:17.613608 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:44:17.613670 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:44:17.613772 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:44:17.614515 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:44:17.615277 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:44:17.616456 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:44:17.616881 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:44:17.616986 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:44:17.620566 (+ 3580us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:44603: BlockingRecv error: recv got EOF from 127.0.97.1:44603 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Jun 19 16:44:18 dist-test-slave-wvz7 krb5kdc[8621](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:44:18 dist-test-slave-wvz7 krb5kdc[8621](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 16:44:18.049952 9637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:44:18.038762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46703 (local address 127.0.97.2:46309)
0619 16:44:18.039256 (+ 494us) negotiation.cc:107] Waiting for socket to connect
0619 16:44:18.039272 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 16:44:18.039379 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:44:18.039716 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:44:18.039720 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:44:18.040065 (+ 345us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:44:18.040251 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:44:18.040260 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:44:18.041292 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:44:18.041296 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:44:18.042381 (+ 1085us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:44:18.042394 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:44:18.043264 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:44:18.043269 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:44:18.043443 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:44:18.046982 (+ 3539us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:44:18.047000 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:44:18.049632 (+ 2632us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:46703: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":383,"thread_start_us":231,"threads_started":1}
W20260619 16:44:18.050062 9359 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:44:18.038978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:46309 (local address 127.0.97.62:46703)
0619 16:44:18.039145 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0619 16:44:18.039150 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:44:18.039339 (+ 189us) server_negotiation.cc:407] Connection header received
0619 16:44:18.039481 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:44:18.039485 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:44:18.039543 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:44:18.039630 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:44:18.040387 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:44:18.041157 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:44:18.042520 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:44:18.043088 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:44:18.043189 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:44:18.049862 (+ 6673us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:46309: BlockingRecv error: recv got EOF from 127.0.97.2:46309 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Jun 19 16:44:18 dist-test-slave-wvz7 krb5kdc[8621](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:44:18 dist-test-slave-wvz7 krb5kdc[8621](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 16:44:18.497900 9638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:44:18.490102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46703 (local address 127.0.97.3:37171)
0619 16:44:18.490646 (+ 544us) negotiation.cc:107] Waiting for socket to connect
0619 16:44:18.490661 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 16:44:18.490769 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:44:18.491057 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:44:18.491060 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:44:18.491423 (+ 363us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:44:18.491589 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:44:18.491597 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:44:18.492594 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:44:18.492598 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:44:18.493506 (+ 908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:44:18.493517 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:44:18.494282 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:44:18.494287 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:44:18.494419 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:44:18.494949 (+ 530us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:44:18.494967 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:44:18.497612 (+ 2645us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:46703: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":430,"thread_start_us":316,"threads_started":1}
W20260619 16:44:18.498019 9359 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:44:18.490134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:37171 (local address 127.0.97.62:46703)
0619 16:44:18.490312 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0619 16:44:18.490318 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:44:18.490738 (+ 420us) server_negotiation.cc:407] Connection header received
0619 16:44:18.490859 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:44:18.490863 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:44:18.490949 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:44:18.491035 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:44:18.491751 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:44:18.492462 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:44:18.493637 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:44:18.494152 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:44:18.494247 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:44:18.497817 (+ 3570us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:37171: BlockingRecv error: recv got EOF from 127.0.97.3:37171 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Jun 19 16:44:18 dist-test-slave-wvz7 krb5kdc[8621](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:44:18 dist-test-slave-wvz7 krb5kdc[8621](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 16:44:18.629390 9639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:44:18.621476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46703 (local address 127.0.97.1:58559)
0619 16:44:18.622020 (+ 544us) negotiation.cc:107] Waiting for socket to connect
0619 16:44:18.622036 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 16:44:18.622145 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:44:18.622454 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:44:18.622458 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:44:18.622873 (+ 415us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:44:18.623018 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:44:18.623026 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:44:18.624020 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:44:18.624024 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:44:18.624931 (+ 907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:44:18.624944 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:44:18.625714 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:44:18.625719 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:44:18.625921 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:44:18.626479 (+ 558us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:44:18.626496 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:44:18.629161 (+ 2665us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:46703: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":447,"thread_start_us":335,"threads_started":1}
W20260619 16:44:18.629554 9359 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:44:18.621524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:58559 (local address 127.0.97.62:46703)
0619 16:44:18.621668 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0619 16:44:18.621673 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:44:18.622135 (+ 462us) server_negotiation.cc:407] Connection header received
0619 16:44:18.622231 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:44:18.622235 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:44:18.622297 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:44:18.622378 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:44:18.623146 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:44:18.623891 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:44:18.625105 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:44:18.625557 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:44:18.625670 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:44:18.629357 (+ 3687us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:58559: BlockingRecv error: recv got EOF from 127.0.97.1:58559 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Jun 19 16:44:19 dist-test-slave-wvz7 krb5kdc[8621](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:44:19 dist-test-slave-wvz7 krb5kdc[8621](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 16:44:19.058595 9640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:44:19.050905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46703 (local address 127.0.97.2:41833)
0619 16:44:19.051364 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0619 16:44:19.051381 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 16:44:19.051496 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:44:19.051816 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:44:19.051820 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:44:19.052174 (+ 354us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:44:19.052328 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:44:19.052336 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:44:19.053325 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:44:19.053329 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:44:19.054272 (+ 943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:44:19.054284 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:44:19.054939 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:44:19.054944 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:44:19.055063 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:44:19.055584 (+ 521us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:44:19.055601 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:44:19.058300 (+ 2699us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:46703: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":364,"thread_start_us":249,"threads_started":1}
W20260619 16:44:19.058760 9359 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:44:19.050998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:41833 (local address 127.0.97.62:46703)
0619 16:44:19.051223 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0619 16:44:19.051229 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:44:19.051476 (+ 247us) server_negotiation.cc:407] Connection header received
0619 16:44:19.051586 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:44:19.051590 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:44:19.051651 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:44:19.051740 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:44:19.052463 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:44:19.053195 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:44:19.054398 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:44:19.054795 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:44:19.054910 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:44:19.058533 (+ 3623us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:41833: BlockingRecv error: recv got EOF from 127.0.97.2:41833 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Jun 19 16:44:19 dist-test-slave-wvz7 krb5kdc[8621](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:44:19 dist-test-slave-wvz7 krb5kdc[8621](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 16:44:19.507256 9641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:44:19.498734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46703 (local address 127.0.97.3:53517)
0619 16:44:19.499240 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0619 16:44:19.499256 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 16:44:19.499404 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:44:19.499766 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:44:19.499771 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:44:19.500192 (+ 421us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:44:19.500350 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:44:19.500359 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:44:19.501423 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:44:19.501428 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:44:19.502636 (+ 1208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:44:19.502653 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:44:19.503518 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:44:19.503524 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:44:19.503686 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:44:19.504237 (+ 551us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:44:19.504261 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:44:19.507005 (+ 2744us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:46703: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":413,"thread_start_us":288,"threads_started":1}
W20260619 16:44:19.507488 9359 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:44:19.498858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:53517 (local address 127.0.97.62:46703)
0619 16:44:19.499070 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0619 16:44:19.499078 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 16:44:19.499341 (+ 263us) server_negotiation.cc:407] Connection header received
0619 16:44:19.499533 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:44:19.499537 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:44:19.499618 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:44:19.499719 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:44:19.500498 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:44:19.501280 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:44:19.502793 (+ 1513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:44:19.503371 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:44:19.503499 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:44:19.507242 (+ 3743us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:53517: BlockingRecv error: recv got EOF from 127.0.97.3:53517 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
Jun 19 16:44:19 dist-test-slave-wvz7 krb5kdc[8621](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:44:19 dist-test-slave-wvz7 krb5kdc[8621](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 16:44:19.638525 9642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:44:19.630285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46703 (local address 127.0.97.1:51807)
0619 16:44:19.630865 (+ 580us) negotiation.cc:107] Waiting for socket to connect
0619 16:44:19.630881 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 16:44:19.630991 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:44:19.631276 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:44:19.631279 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:44:19.631643 (+ 364us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:44:19.631841 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:44:19.631850 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:44:19.632915 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:44:19.632920 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:44:19.634169 (+ 1249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:44:19.634181 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:44:19.634885 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:44:19.634891 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:44:19.635047 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:44:19.635632 (+ 585us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:44:19.635650 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:44:19.638294 (+ 2644us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:46703: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":454,"thread_start_us":243,"threads_started":1}
W20260619 16:44:19.638638 9359 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:44:19.630411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:51807 (local address 127.0.97.62:46703)
0619 16:44:19.630576 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0619 16:44:19.630581 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:44:19.630944 (+ 363us) server_negotiation.cc:407] Connection header received
0619 16:44:19.631084 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:44:19.631088 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:44:19.631146 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:44:19.631229 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:44:19.631956 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:44:19.632728 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:44:19.634300 (+ 1572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:44:19.634742 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:44:19.634849 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:44:19.638466 (+ 3617us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:51807: BlockingRecv error: recv got EOF from 127.0.97.1:51807 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Jun 19 16:44:20 dist-test-slave-wvz7 krb5kdc[8621](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:44:20 dist-test-slave-wvz7 krb5kdc[8621](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 16:44:20.068156 9643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:44:20.059571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46703 (local address 127.0.97.2:36137)
0619 16:44:20.060146 (+ 575us) negotiation.cc:107] Waiting for socket to connect
0619 16:44:20.060162 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 16:44:20.060287 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:44:20.060594 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:44:20.060597 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:44:20.061004 (+ 407us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:44:20.061170 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:44:20.061178 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:44:20.062457 (+ 1279us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:44:20.062467 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:44:20.063718 (+ 1251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:44:20.063739 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:44:20.064405 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:44:20.064412 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:44:20.064563 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:44:20.065101 (+ 538us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:44:20.065119 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:44:20.067833 (+ 2714us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:46703: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":448,"thread_start_us":267,"threads_started":1}
W20260619 16:44:20.068274 9359 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:44:20.059694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:36137 (local address 127.0.97.62:46703)
0619 16:44:20.059855 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0619 16:44:20.059860 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:44:20.060271 (+ 411us) server_negotiation.cc:407] Connection header received
0619 16:44:20.060405 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:44:20.060408 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:44:20.060474 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:44:20.060566 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:44:20.061305 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:44:20.062147 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:44:20.063853 (+ 1706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:44:20.064259 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:44:20.064366 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:44:20.068038 (+ 3672us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:36137: BlockingRecv error: recv got EOF from 127.0.97.2:36137 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Jun 19 16:44:20 dist-test-slave-wvz7 krb5kdc[8621](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:44:20 dist-test-slave-wvz7 krb5kdc[8621](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 16:44:20.515882 9644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:44:20.508087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46703 (local address 127.0.97.3:33963)
0619 16:44:20.508579 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0619 16:44:20.508594 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 16:44:20.508702 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:44:20.509024 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:44:20.509027 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:44:20.509400 (+ 373us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:44:20.509551 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:44:20.509559 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:44:20.510521 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:44:20.510524 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:44:20.511416 (+ 892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:44:20.511427 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:44:20.512064 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:44:20.512068 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:44:20.512171 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:44:20.512661 (+ 490us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:44:20.512677 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:44:20.515579 (+ 2902us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:46703: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":379,"thread_start_us":217,"threads_started":1}
W20260619 16:44:20.516034 9359 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:44:20.508146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:33963 (local address 127.0.97.62:46703)
0619 16:44:20.508332 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0619 16:44:20.508339 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:44:20.508668 (+ 329us) server_negotiation.cc:407] Connection header received
0619 16:44:20.508827 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:44:20.508831 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:44:20.508906 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:44:20.508991 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:44:20.509656 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:44:20.510413 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:44:20.511536 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:44:20.511952 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:44:20.512044 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:44:20.515825 (+ 3781us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:33963: BlockingRecv error: recv got EOF from 127.0.97.3:33963 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Jun 19 16:44:20 dist-test-slave-wvz7 krb5kdc[8621](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:44:20 dist-test-slave-wvz7 krb5kdc[8621](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 16:44:20.647395 9645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:44:20.639391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46703 (local address 127.0.97.1:52915)
0619 16:44:20.639810 (+ 419us) negotiation.cc:107] Waiting for socket to connect
0619 16:44:20.639829 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 16:44:20.639962 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:44:20.640260 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:44:20.640264 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:44:20.640664 (+ 400us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:44:20.640806 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:44:20.640814 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:44:20.641827 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:44:20.641833 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:44:20.643056 (+ 1223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:44:20.643071 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:44:20.643774 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:44:20.643780 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:44:20.643956 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:44:20.644557 (+ 601us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:44:20.644575 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:44:20.647174 (+ 2599us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:46703: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":323,"thread_start_us":201,"threads_started":1}
W20260619 16:44:20.647567 9359 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:44:20.639598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:52915 (local address 127.0.97.62:46703)
0619 16:44:20.639809 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0619 16:44:20.639815 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:44:20.639893 (+ 78us) server_negotiation.cc:407] Connection header received
0619 16:44:20.640075 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:44:20.640080 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:44:20.640140 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:44:20.640215 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:44:20.640937 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:44:20.641674 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:44:20.643195 (+ 1521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:44:20.643610 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:44:20.643720 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:44:20.647344 (+ 3624us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:52915: BlockingRecv error: recv got EOF from 127.0.97.1:52915 (error 108)
Metrics: {"server-negotiator.queue_time_us":85}
Jun 19 16:44:21 dist-test-slave-wvz7 krb5kdc[8621](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:44:21 dist-test-slave-wvz7 krb5kdc[8621](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 16:44:21.076756 9646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:44:21.069042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46703 (local address 127.0.97.2:39455)
0619 16:44:21.069515 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0619 16:44:21.069529 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 16:44:21.069658 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:44:21.069983 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:44:21.069988 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:44:21.070366 (+ 378us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:44:21.070505 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:44:21.070514 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:44:21.071647 (+ 1133us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:44:21.071651 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:44:21.072535 (+ 884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:44:21.072547 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:44:21.073242 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:44:21.073248 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:44:21.073402 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:44:21.073960 (+ 558us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:44:21.073976 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:44:21.076461 (+ 2485us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:46703: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":382,"thread_start_us":264,"threads_started":1}
W20260619 16:44:21.076860 9359 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:44:21.069128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:39455 (local address 127.0.97.62:46703)
0619 16:44:21.069328 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0619 16:44:21.069335 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:44:21.069613 (+ 278us) server_negotiation.cc:407] Connection header received
0619 16:44:21.069783 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:44:21.069788 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:44:21.069860 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:44:21.069940 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:44:21.070647 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:44:21.071513 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:44:21.072666 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:44:21.073084 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:44:21.073187 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:44:21.076666 (+ 3479us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:39455: BlockingRecv error: recv got EOF from 127.0.97.2:39455 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Jun 19 16:44:21 dist-test-slave-wvz7 krb5kdc[8621](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:44:21 dist-test-slave-wvz7 krb5kdc[8621](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 16:44:21.524271 9647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:44:21.516778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46703 (local address 127.0.97.3:54019)
0619 16:44:21.517299 (+ 521us) negotiation.cc:107] Waiting for socket to connect
0619 16:44:21.517313 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 16:44:21.517442 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:44:21.517794 (+ 352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:44:21.517800 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:44:21.518180 (+ 380us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:44:21.518323 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:44:21.518332 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:44:21.519278 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:44:21.519284 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:44:21.520193 (+ 909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:44:21.520204 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:44:21.520918 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:44:21.520925 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:44:21.521085 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:44:21.521586 (+ 501us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:44:21.521602 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:44:21.524047 (+ 2445us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:46703: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":412,"thread_start_us":284,"threads_started":1}
W20260619 16:44:21.524430 9359 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:44:21.516877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:54019 (local address 127.0.97.62:46703)
0619 16:44:21.517164 (+ 287us) server_negotiation.cc:206] Beginning negotiation
0619 16:44:21.517170 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:44:21.517389 (+ 219us) server_negotiation.cc:407] Connection header received
0619 16:44:21.517550 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:44:21.517555 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:44:21.517625 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:44:21.517703 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:44:21.518453 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:44:21.519139 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:44:21.520314 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:44:21.520759 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:44:21.520865 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:44:21.524228 (+ 3363us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:54019: BlockingRecv error: recv got EOF from 127.0.97.3:54019 (error 108)
Metrics: {"server-negotiator.queue_time_us":136}
Jun 19 16:44:21 dist-test-slave-wvz7 krb5kdc[8621](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:44:21 dist-test-slave-wvz7 krb5kdc[8621](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 16:44:21.656077 9648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:44:21.648350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46703 (local address 127.0.97.1:58475)
0619 16:44:21.648793 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0619 16:44:21.648808 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 16:44:21.648933 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:44:21.649273 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:44:21.649275 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:44:21.649632 (+ 357us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:44:21.649851 (+ 219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:44:21.649859 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:44:21.650786 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:44:21.650790 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:44:21.651811 (+ 1021us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:44:21.651824 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:44:21.652470 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:44:21.652474 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:44:21.652592 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:44:21.653106 (+ 514us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:44:21.653121 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:44:21.655768 (+ 2647us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:46703: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":355,"thread_start_us":216,"threads_started":1}
W20260619 16:44:21.656133 9359 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:44:21.648415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:58475 (local address 127.0.97.62:46703)
0619 16:44:21.648613 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0619 16:44:21.648618 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:44:21.648877 (+ 259us) server_negotiation.cc:407] Connection header received
0619 16:44:21.649097 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:44:21.649101 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:44:21.649163 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:44:21.649241 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:44:21.649953 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:44:21.650642 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:44:21.651937 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:44:21.652332 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:44:21.652416 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:44:21.655959 (+ 3543us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:58475: BlockingRecv error: recv got EOF from 127.0.97.1:58475 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Jun 19 16:44:22 dist-test-slave-wvz7 krb5kdc[8621](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:44:22 dist-test-slave-wvz7 krb5kdc[8621](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 16:44:22.085096 9649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:44:22.077564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46703 (local address 127.0.97.2:35777)
0619 16:44:22.078053 (+ 489us) negotiation.cc:107] Waiting for socket to connect
0619 16:44:22.078068 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 16:44:22.078198 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:44:22.078493 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:44:22.078497 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:44:22.078856 (+ 359us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:44:22.079011 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:44:22.079019 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:44:22.079970 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:44:22.079973 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:44:22.080873 (+ 900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:44:22.080885 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:44:22.081525 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:44:22.081530 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:44:22.081656 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:44:22.082256 (+ 600us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:44:22.082279 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:44:22.084851 (+ 2572us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:46703: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":387,"thread_start_us":257,"threads_started":1}
W20260619 16:44:22.085289 9359 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:44:22.077630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:35777 (local address 127.0.97.62:46703)
0619 16:44:22.077854 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0619 16:44:22.077860 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:44:22.078178 (+ 318us) server_negotiation.cc:407] Connection header received
0619 16:44:22.078299 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:44:22.078303 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:44:22.078363 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:44:22.078448 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:44:22.079111 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:44:22.079865 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:44:22.080980 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:44:22.081368 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:44:22.081460 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:44:22.085067 (+ 3607us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:35777: BlockingRecv error: recv got EOF from 127.0.97.2:35777 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Jun 19 16:44:22 dist-test-slave-wvz7 krb5kdc[8621](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:44:22 dist-test-slave-wvz7 krb5kdc[8621](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 16:44:22.532639 9650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:44:22.525084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46703 (local address 127.0.97.3:44311)
0619 16:44:22.525591 (+ 507us) negotiation.cc:107] Waiting for socket to connect
0619 16:44:22.525607 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 16:44:22.525720 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:44:22.526010 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:44:22.526013 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:44:22.526370 (+ 357us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:44:22.526530 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:44:22.526538 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:44:22.527489 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:44:22.527493 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:44:22.528393 (+ 900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:44:22.528404 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:44:22.529043 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:44:22.529047 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:44:22.529157 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:44:22.529677 (+ 520us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:44:22.529694 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:44:22.532411 (+ 2717us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:46703: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":418,"thread_start_us":281,"threads_started":1}
W20260619 16:44:22.532833 9359 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:44:22.525137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:44311 (local address 127.0.97.62:46703)
0619 16:44:22.525287 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0619 16:44:22.525291 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:44:22.525699 (+ 408us) server_negotiation.cc:407] Connection header received
0619 16:44:22.525828 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:44:22.525831 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:44:22.525903 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:44:22.525984 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:44:22.526673 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:44:22.527362 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:44:22.528518 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:44:22.528933 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:44:22.529024 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:44:22.532592 (+ 3568us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:44311: BlockingRecv error: recv got EOF from 127.0.97.3:44311 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Jun 19 16:44:22 dist-test-slave-wvz7 krb5kdc[8621](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:44:22 dist-test-slave-wvz7 krb5kdc[8621](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 16:44:22.664826 9651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:44:22.656934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46703 (local address 127.0.97.1:44223)
0619 16:44:22.657448 (+ 514us) negotiation.cc:107] Waiting for socket to connect
0619 16:44:22.657466 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 16:44:22.657574 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:44:22.657863 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:44:22.657866 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:44:22.658198 (+ 332us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:44:22.658337 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:44:22.658352 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:44:22.659270 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:44:22.659274 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:44:22.660371 (+ 1097us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:44:22.660386 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:44:22.661293 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:44:22.661300 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:44:22.661448 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:44:22.661995 (+ 547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:44:22.662010 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:44:22.664496 (+ 2486us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:46703: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":413,"thread_start_us":313,"threads_started":1}
W20260619 16:44:22.664904 9359 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:44:22.657046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:44223 (local address 127.0.97.62:46703)
0619 16:44:22.657191 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0619 16:44:22.657196 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:44:22.657532 (+ 336us) server_negotiation.cc:407] Connection header received
0619 16:44:22.657655 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:44:22.657659 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:44:22.657714 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:44:22.657813 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:44:22.658454 (+ 641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:44:22.659166 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:44:22.660531 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:44:22.661138 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:44:22.661271 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:44:22.664689 (+ 3418us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:44223: BlockingRecv error: recv got EOF from 127.0.97.1:44223 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Jun 19 16:44:23 dist-test-slave-wvz7 krb5kdc[8621](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:44:23 dist-test-slave-wvz7 krb5kdc[8621](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 16:44:23.096410 9652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:44:23.085906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46703 (local address 127.0.97.2:44305)
0619 16:44:23.086335 (+ 429us) negotiation.cc:107] Waiting for socket to connect
0619 16:44:23.086351 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 16:44:23.086456 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:44:23.086711 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:44:23.086714 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:44:23.087051 (+ 337us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:44:23.087200 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:44:23.087209 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:44:23.088133 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:44:23.088137 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:44:23.089022 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:44:23.089032 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:44:23.089643 (+ 611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:44:23.089647 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:44:23.089799 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:44:23.093526 (+ 3727us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:44:23.093543 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:44:23.096189 (+ 2646us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:46703: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":328,"thread_start_us":230,"threads_started":1}
W20260619 16:44:23.096643 9359 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:44:23.085992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:44305 (local address 127.0.97.62:46703)
0619 16:44:23.086200 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0619 16:44:23.086206 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:44:23.086418 (+ 212us) server_negotiation.cc:407] Connection header received
0619 16:44:23.086540 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:44:23.086543 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:44:23.086599 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:44:23.086673 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:44:23.087307 (+ 634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:44:23.088019 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:44:23.089120 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:44:23.089518 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:44:23.089603 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:44:23.096379 (+ 6776us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:44305: BlockingRecv error: recv got EOF from 127.0.97.2:44305 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
Jun 19 16:44:23 dist-test-slave-wvz7 krb5kdc[8621](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:44:23 dist-test-slave-wvz7 krb5kdc[8621](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 16:44:23.541611 9653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:44:23.533548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46703 (local address 127.0.97.3:56815)
0619 16:44:23.534036 (+ 488us) negotiation.cc:107] Waiting for socket to connect
0619 16:44:23.534054 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 16:44:23.534201 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:44:23.534534 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:44:23.534538 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:44:23.534899 (+ 361us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:44:23.535036 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:44:23.535045 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:44:23.536246 (+ 1201us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:44:23.536251 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:44:23.537291 (+ 1040us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:44:23.537303 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:44:23.537995 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:44:23.538000 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:44:23.538121 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:44:23.538716 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:44:23.538733 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:44:23.541301 (+ 2568us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:46703: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":369,"thread_start_us":267,"threads_started":1}
W20260619 16:44:23.541725 9359 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:44:23.533813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:56815 (local address 127.0.97.62:46703)
0619 16:44:23.534039 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0619 16:44:23.534045 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:44:23.534131 (+ 86us) server_negotiation.cc:407] Connection header received
0619 16:44:23.534316 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:44:23.534320 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:44:23.534408 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:44:23.534489 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:44:23.535215 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:44:23.536107 (+ 892us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:44:23.537426 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:44:23.537864 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:44:23.537954 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:44:23.541504 (+ 3550us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:56815: BlockingRecv error: recv got EOF from 127.0.97.3:56815 (error 108)
Metrics: {"server-negotiator.queue_time_us":103}
Jun 19 16:44:23 dist-test-slave-wvz7 krb5kdc[8621](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:44:23 dist-test-slave-wvz7 krb5kdc[8621](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 16:44:23.673434 9654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:44:23.665649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46703 (local address 127.0.97.1:53933)
0619 16:44:23.666167 (+ 518us) negotiation.cc:107] Waiting for socket to connect
0619 16:44:23.666186 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 16:44:23.666349 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:44:23.666656 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:44:23.666661 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:44:23.667173 (+ 512us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:44:23.667381 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:44:23.667394 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:44:23.668339 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:44:23.668343 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:44:23.669232 (+ 889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:44:23.669250 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:44:23.669952 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:44:23.669956 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:44:23.670068 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:44:23.670626 (+ 558us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:44:23.670642 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:44:23.673205 (+ 2563us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:46703: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":413,"thread_start_us":320,"threads_started":1}
W20260619 16:44:23.673583 9359 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:44:23.665775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:53933 (local address 127.0.97.62:46703)
0619 16:44:23.665960 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0619 16:44:23.665966 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:44:23.666272 (+ 306us) server_negotiation.cc:407] Connection header received
0619 16:44:23.666461 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:44:23.666465 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:44:23.666524 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:44:23.666595 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:44:23.667503 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:44:23.668234 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:44:23.669361 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:44:23.669844 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:44:23.669933 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:44:23.673394 (+ 3461us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:53933: BlockingRecv error: recv got EOF from 127.0.97.1:53933 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
Jun 19 16:44:24 dist-test-slave-wvz7 krb5kdc[8621](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:44:24 dist-test-slave-wvz7 krb5kdc[8621](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 16:44:24.109054 9655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:44:24.097250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46703 (local address 127.0.97.2:40399)
0619 16:44:24.097656 (+ 406us) negotiation.cc:107] Waiting for socket to connect
0619 16:44:24.097671 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 16:44:24.097866 (+ 195us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:44:24.098147 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:44:24.098151 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:44:24.098522 (+ 371us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:44:24.098682 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:44:24.098691 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:44:24.099667 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:44:24.099671 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:44:24.100587 (+ 916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:44:24.100597 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:44:24.101203 (+ 606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:44:24.101207 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:44:24.101310 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:44:24.106184 (+ 4874us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:44:24.106201 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:44:24.108826 (+ 2625us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:46703: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":313,"thread_start_us":204,"threads_started":1}
W20260619 16:44:24.109182 9359 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:44:24.097322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:40399 (local address 127.0.97.62:46703)
0619 16:44:24.097515 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0619 16:44:24.097520 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:44:24.097796 (+ 276us) server_negotiation.cc:407] Connection header received
0619 16:44:24.097975 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:44:24.097978 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:44:24.098037 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:44:24.098111 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:44:24.098814 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:44:24.099547 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:44:24.100688 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:44:24.101100 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:44:24.101182 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:44:24.108998 (+ 7816us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:40399: BlockingRecv error: recv got EOF from 127.0.97.2:40399 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Jun 19 16:44:24 dist-test-slave-wvz7 krb5kdc[8621](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:44:24 dist-test-slave-wvz7 krb5kdc[8621](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 16:44:24.550437 9656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:44:24.542523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46703 (local address 127.0.97.3:59865)
0619 16:44:24.542990 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0619 16:44:24.543009 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 16:44:24.543141 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:44:24.543427 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:44:24.543431 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:44:24.543918 (+ 487us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:44:24.544131 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:44:24.544143 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:44:24.545083 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:44:24.545087 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:44:24.546070 (+ 983us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:44:24.546085 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:44:24.546825 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:44:24.546830 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:44:24.547003 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:44:24.547648 (+ 645us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:44:24.547671 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:44:24.550214 (+ 2543us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:46703: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":355,"thread_start_us":234,"threads_started":1}
W20260619 16:44:24.550567 9359 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:44:24.542638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:59865 (local address 127.0.97.62:46703)
0619 16:44:24.542817 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0619 16:44:24.542823 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:44:24.543098 (+ 275us) server_negotiation.cc:407] Connection header received
0619 16:44:24.543244 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:44:24.543248 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:44:24.543307 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:44:24.543381 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:44:24.544262 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:44:24.544969 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:44:24.546187 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:44:24.546674 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:44:24.546789 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:44:24.550379 (+ 3590us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:59865: BlockingRecv error: recv got EOF from 127.0.97.3:59865 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Jun 19 16:44:24 dist-test-slave-wvz7 krb5kdc[8621](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:44:24 dist-test-slave-wvz7 krb5kdc[8621](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 16:44:24.682042 9657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:44:24.674217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46703 (local address 127.0.97.1:44563)
0619 16:44:24.674711 (+ 494us) negotiation.cc:107] Waiting for socket to connect
0619 16:44:24.674725 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 16:44:24.674838 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:44:24.675140 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:44:24.675143 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:44:24.675512 (+ 369us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:44:24.675667 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:44:24.675676 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:44:24.676781 (+ 1105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:44:24.676785 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:44:24.677853 (+ 1068us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:44:24.677865 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:44:24.678670 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:44:24.678675 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:44:24.678804 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:44:24.679352 (+ 548us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:44:24.679368 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:44:24.681822 (+ 2454us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:46703: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":402,"thread_start_us":277,"threads_started":1}
W20260619 16:44:24.682235 9359 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:44:24.674273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:44563 (local address 127.0.97.62:46703)
0619 16:44:24.674428 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0619 16:44:24.674434 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:44:24.674825 (+ 391us) server_negotiation.cc:407] Connection header received
0619 16:44:24.674928 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:44:24.674932 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:44:24.674996 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:44:24.675090 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:44:24.675814 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:44:24.676662 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:44:24.677964 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:44:24.678543 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:44:24.678631 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:44:24.682009 (+ 3378us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:44563: BlockingRecv error: recv got EOF from 127.0.97.1:44563 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Jun 19 16:44:25 dist-test-slave-wvz7 krb5kdc[8621](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:44:25 dist-test-slave-wvz7 krb5kdc[8621](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 16:44:25.118474 9658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:44:25.109936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46703 (local address 127.0.97.2:42145)
0619 16:44:25.110435 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0619 16:44:25.110450 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 16:44:25.110573 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:44:25.110932 (+ 359us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:44:25.110936 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:44:25.111307 (+ 371us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:44:25.111458 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:44:25.111466 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:44:25.112407 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:44:25.112411 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:44:25.113512 (+ 1101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:44:25.113523 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:44:25.114246 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:44:25.114251 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:44:25.114434 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:44:25.115038 (+ 604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:44:25.115054 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:44:25.118110 (+ 3056us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:46703: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":408,"thread_start_us":280,"threads_started":1}
W20260619 16:44:25.118486 9359 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:44:25.109995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:42145 (local address 127.0.97.62:46703)
0619 16:44:25.110217 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0619 16:44:25.110224 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:44:25.110524 (+ 300us) server_negotiation.cc:407] Connection header received
0619 16:44:25.110736 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:44:25.110742 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:44:25.110803 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:44:25.110886 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:44:25.111587 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:44:25.112282 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:44:25.113634 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:44:25.114119 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:44:25.114238 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:44:25.118302 (+ 4064us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:42145: BlockingRecv error: recv got EOF from 127.0.97.2:42145 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Jun 19 16:44:25 dist-test-slave-wvz7 krb5kdc[8621](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:44:25 dist-test-slave-wvz7 krb5kdc[8621](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 16:44:25.560182 9659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:44:25.551202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46703 (local address 127.0.97.3:35531)
0619 16:44:25.551716 (+ 514us) negotiation.cc:107] Waiting for socket to connect
0619 16:44:25.551732 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 16:44:25.551831 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:44:25.552134 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:44:25.552137 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:44:25.552508 (+ 371us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:44:25.552672 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:44:25.552680 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:44:25.554097 (+ 1417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:44:25.554101 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:44:25.555049 (+ 948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:44:25.555060 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:44:25.555964 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:44:25.555968 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:44:25.556101 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:44:25.556745 (+ 644us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:44:25.556761 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:44:25.559852 (+ 3091us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:46703: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":421,"thread_start_us":297,"threads_started":1}
W20260619 16:44:25.560298 9359 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:44:25.551274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:35531 (local address 127.0.97.62:46703)
0619 16:44:25.551457 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0619 16:44:25.551463 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:44:25.551807 (+ 344us) server_negotiation.cc:407] Connection header received
0619 16:44:25.551926 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:44:25.551931 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:44:25.552009 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:44:25.552103 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:44:25.552787 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:44:25.553959 (+ 1172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:44:25.555178 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:44:25.555845 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:44:25.555967 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:44:25.560074 (+ 4107us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:35531: BlockingRecv error: recv got EOF from 127.0.97.3:35531 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Jun 19 16:44:25 dist-test-slave-wvz7 krb5kdc[8621](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:44:25 dist-test-slave-wvz7 krb5kdc[8621](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 16:44:25.690418 9660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:44:25.682864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46703 (local address 127.0.97.1:47505)
0619 16:44:25.683330 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0619 16:44:25.683345 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 16:44:25.683467 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:44:25.683776 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:44:25.683781 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:44:25.684155 (+ 374us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:44:25.684322 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:44:25.684331 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:44:25.685267 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:44:25.685273 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:44:25.686187 (+ 914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:44:25.686199 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:44:25.686872 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:44:25.686879 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:44:25.687024 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:44:25.687525 (+ 501us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:44:25.687541 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:44:25.690182 (+ 2641us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:46703: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":378,"thread_start_us":227,"threads_started":1}
W20260619 16:44:25.690608 9359 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:44:25.682924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:47505 (local address 127.0.97.62:46703)
0619 16:44:25.683111 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0619 16:44:25.683117 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:44:25.683420 (+ 303us) server_negotiation.cc:407] Connection header received
0619 16:44:25.683571 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:44:25.683576 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:44:25.683636 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:44:25.683709 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:44:25.684454 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:44:25.685147 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:44:25.686319 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:44:25.686720 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:44:25.686821 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:44:25.690380 (+ 3559us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:47505: BlockingRecv error: recv got EOF from 127.0.97.1:47505 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Jun 19 16:44:26 dist-test-slave-wvz7 krb5kdc[8621](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:44:26 dist-test-slave-wvz7 krb5kdc[8621](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 16:44:26.127662 9661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:44:26.119316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46703 (local address 127.0.97.2:40337)
0619 16:44:26.120003 (+ 687us) negotiation.cc:107] Waiting for socket to connect
0619 16:44:26.120022 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 16:44:26.120150 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:44:26.120436 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:44:26.120440 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:44:26.120846 (+ 406us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:44:26.121041 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:44:26.121051 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:44:26.122069 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:44:26.122073 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:44:26.123021 (+ 948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:44:26.123033 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:44:26.123754 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:44:26.123759 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:44:26.123868 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:44:26.124468 (+ 600us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:44:26.124485 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:44:26.127375 (+ 2890us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:46703: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":578,"thread_start_us":320,"threads_started":1}
W20260619 16:44:26.127802 9359 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:44:26.119514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:40337 (local address 127.0.97.62:46703)
0619 16:44:26.119770 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0619 16:44:26.119776 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:44:26.120132 (+ 356us) server_negotiation.cc:407] Connection header received
0619 16:44:26.120240 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:44:26.120244 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:44:26.120304 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:44:26.120386 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:44:26.121182 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:44:26.121933 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:44:26.123162 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:44:26.123576 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:44:26.123669 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:44:26.127576 (+ 3907us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:40337: BlockingRecv error: recv got EOF from 127.0.97.2:40337 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
Jun 19 16:44:26 dist-test-slave-wvz7 krb5kdc[8621](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:44:26 dist-test-slave-wvz7 krb5kdc[8621](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 16:44:26.569264 9662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:44:26.561204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46703 (local address 127.0.97.3:33419)
0619 16:44:26.561623 (+ 419us) negotiation.cc:107] Waiting for socket to connect
0619 16:44:26.561639 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 16:44:26.561819 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:44:26.562134 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:44:26.562137 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:44:26.562500 (+ 363us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:44:26.562658 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:44:26.562696 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:44:26.563957 (+ 1261us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:44:26.563962 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:44:26.564936 (+ 974us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:44:26.564949 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:44:26.565613 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:44:26.565618 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:44:26.565816 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:44:26.566336 (+ 520us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:44:26.566352 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:44:26.569028 (+ 2676us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:46703: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":322,"thread_start_us":204,"threads_started":1}
W20260619 16:44:26.569413 9359 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:44:26.561299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:33419 (local address 127.0.97.62:46703)
0619 16:44:26.561507 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0619 16:44:26.561513 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:44:26.561762 (+ 249us) server_negotiation.cc:407] Connection header received
0619 16:44:26.561919 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:44:26.561922 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:44:26.562010 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:44:26.562094 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:44:26.562826 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:44:26.563822 (+ 996us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:44:26.565075 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:44:26.565483 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:44:26.565585 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:44:26.569221 (+ 3636us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:33419: BlockingRecv error: recv got EOF from 127.0.97.3:33419 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
Jun 19 16:44:26 dist-test-slave-wvz7 krb5kdc[8621](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:44:26 dist-test-slave-wvz7 krb5kdc[8621](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 16:44:26.699520 9663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:44:26.691250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46703 (local address 127.0.97.1:46181)
0619 16:44:26.691747 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0619 16:44:26.691762 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 16:44:26.691890 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:44:26.692197 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:44:26.692202 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:44:26.692670 (+ 468us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:44:26.692844 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:44:26.692856 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:44:26.693961 (+ 1105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:44:26.693967 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:44:26.694898 (+ 931us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:44:26.694910 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:44:26.695596 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:44:26.695601 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:44:26.695770 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:44:26.696396 (+ 626us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:44:26.696414 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:44:26.699194 (+ 2780us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:46703: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":372,"thread_start_us":248,"threads_started":1}
W20260619 16:44:26.699645 9359 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:44:26.691363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:46181 (local address 127.0.97.62:46703)
0619 16:44:26.691655 (+ 292us) server_negotiation.cc:206] Beginning negotiation
0619 16:44:26.691662 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:44:26.691836 (+ 174us) server_negotiation.cc:407] Connection header received
0619 16:44:26.692001 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:44:26.692005 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:44:26.692059 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:44:26.692130 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:44:26.692958 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:44:26.693802 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:44:26.695019 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:44:26.695479 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:44:26.695569 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:44:26.699425 (+ 3856us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:46181: BlockingRecv error: recv got EOF from 127.0.97.1:46181 (error 108)
Metrics: {"server-negotiator.queue_time_us":142}
Jun 19 16:44:27 dist-test-slave-wvz7 krb5kdc[8621](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:44:27 dist-test-slave-wvz7 krb5kdc[8621](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 16:44:27.136583 9664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:44:27.128500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46703 (local address 127.0.97.2:52451)
0619 16:44:27.128951 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0619 16:44:27.128969 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 16:44:27.129081 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:44:27.129366 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:44:27.129370 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:44:27.129765 (+ 395us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:44:27.129916 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:44:27.129924 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:44:27.130969 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:44:27.130973 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:44:27.131872 (+ 899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:44:27.131884 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:44:27.132525 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:44:27.132528 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:44:27.132662 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:44:27.133210 (+ 548us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:44:27.133226 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:44:27.136345 (+ 3119us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:46703: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":343,"thread_start_us":241,"threads_started":1}
W20260619 16:44:27.136722 9359 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:44:27.128629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:52451 (local address 127.0.97.62:46703)
0619 16:44:27.128771 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0619 16:44:27.128776 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:44:27.129032 (+ 256us) server_negotiation.cc:407] Connection header received
0619 16:44:27.129164 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:44:27.129168 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:44:27.129224 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:44:27.129304 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:44:27.130029 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:44:27.130839 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:44:27.131998 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:44:27.132418 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:44:27.132508 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:44:27.136513 (+ 4005us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:52451: BlockingRecv error: recv got EOF from 127.0.97.2:52451 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Jun 19 16:44:27 dist-test-slave-wvz7 krb5kdc[8621](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:44:27 dist-test-slave-wvz7 krb5kdc[8621](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 16:44:27.577940 9665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:44:27.570048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46703 (local address 127.0.97.3:46661)
0619 16:44:27.570510 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0619 16:44:27.570526 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 16:44:27.570638 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:44:27.570889 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:44:27.570893 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:44:27.571201 (+ 308us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:44:27.571360 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:44:27.571369 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:44:27.572330 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:44:27.572334 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:44:27.573242 (+ 908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:44:27.573255 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:44:27.574049 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:44:27.574055 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:44:27.574217 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:44:27.574854 (+ 637us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:44:27.574871 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:44:27.577590 (+ 2719us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:46703: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":359,"thread_start_us":235,"threads_started":1}
W20260619 16:44:27.578065 9359 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:44:27.570153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:46661 (local address 127.0.97.62:46703)
0619 16:44:27.570323 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0619 16:44:27.570328 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:44:27.570588 (+ 260us) server_negotiation.cc:407] Connection header received
0619 16:44:27.570720 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:44:27.570723 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:44:27.570780 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:44:27.570861 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:44:27.571465 (+ 604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:44:27.572178 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:44:27.573447 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:44:27.573892 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:44:27.574012 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:44:27.577861 (+ 3849us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:46661: BlockingRecv error: recv got EOF from 127.0.97.3:46661 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Jun 19 16:44:27 dist-test-slave-wvz7 krb5kdc[8621](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:44:27 dist-test-slave-wvz7 krb5kdc[8621](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 16:44:27.708918 9666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:44:27.700520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46703 (local address 127.0.97.1:39091)
0619 16:44:27.701166 (+ 646us) negotiation.cc:107] Waiting for socket to connect
0619 16:44:27.701182 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 16:44:27.701303 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:44:27.701607 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:44:27.701610 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:44:27.702050 (+ 440us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:44:27.702239 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:44:27.702250 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:44:27.703372 (+ 1122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:44:27.703378 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:44:27.704561 (+ 1183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:44:27.704579 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:44:27.705221 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:44:27.705226 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:44:27.705348 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:44:27.705958 (+ 610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:44:27.705976 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:44:27.708581 (+ 2605us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:46703: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":550,"thread_start_us":437,"threads_started":1}
W20260619 16:44:27.709005 9359 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:44:27.700590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:39091 (local address 127.0.97.62:46703)
0619 16:44:27.700833 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0619 16:44:27.700838 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:44:27.701263 (+ 425us) server_negotiation.cc:407] Connection header received
0619 16:44:27.701406 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:44:27.701409 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:44:27.701474 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:44:27.701540 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:44:27.702366 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:44:27.703222 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:44:27.704684 (+ 1462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:44:27.705091 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:44:27.705182 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:44:27.708795 (+ 3613us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:39091: BlockingRecv error: recv got EOF from 127.0.97.1:39091 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
Jun 19 16:44:28 dist-test-slave-wvz7 krb5kdc[8621](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:44:28 dist-test-slave-wvz7 krb5kdc[8621](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 16:44:28.147847 9667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:44:28.137589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46703 (local address 127.0.97.2:37373)
0619 16:44:28.138059 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0619 16:44:28.138074 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 16:44:28.138184 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:44:28.138428 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:44:28.138431 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:44:28.138788 (+ 357us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:44:28.138943 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:44:28.138951 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:44:28.139890 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:44:28.139894 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:44:28.140779 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:44:28.140790 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:44:28.141385 (+ 595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:44:28.141389 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:44:28.141495 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:44:28.144863 (+ 3368us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:44:28.144881 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:44:28.147528 (+ 2647us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:46703: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":368,"thread_start_us":272,"threads_started":1}
W20260619 16:44:28.147888 9359 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:44:28.137683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:37373 (local address 127.0.97.62:46703)
0619 16:44:28.137885 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0619 16:44:28.137893 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 16:44:28.138144 (+ 251us) server_negotiation.cc:407] Connection header received
0619 16:44:28.138267 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:44:28.138270 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:44:28.138327 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:44:28.138398 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:44:28.139047 (+ 649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:44:28.139778 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:44:28.140878 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:44:28.141286 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:44:28.141375 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:44:28.147709 (+ 6334us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:37373: BlockingRecv error: recv got EOF from 127.0.97.2:37373 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
I20260619 16:44:28.426584 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu pid 9208
I20260619 16:44:28.437390 9669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:44:28.427501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37512 (local address 127.0.97.1:36879)
0619 16:44:28.427975 (+ 474us) server_negotiation.cc:206] Beginning negotiation
0619 16:44:28.427982 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:44:28.428006 (+ 24us) server_negotiation.cc:407] Connection header received
0619 16:44:28.428060 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:44:28.428065 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:44:28.428283 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:44:28.428434 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:44:28.429847 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:44:28.430841 (+ 994us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:44:28.431930 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:44:28.432461 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:44:28.432551 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:44:28.434779 (+ 2228us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:44:28.434787 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:44:28.434799 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:44:28.434832 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:44:28.436956 (+ 2124us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.97.1@KRBTEST.COM not found in keytab (ticket kvno 2)
0619 16:44:28.437185 (+ 229us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:37512: Request ticket server kudu/127.0.97.1@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":327,"thread_start_us":219,"threads_started":1}
W20260619 16:44:28.437495 9669 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:37512: Request ticket server kudu/127.0.97.1@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260619 16:44:28.437565 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu pid 9208: Not authorized: Client connection negotiation failed: client connection to 127.0.97.1:36879: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.97.1@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260619 16:44:28.437785 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu with pid 9208
I20260619 16:44:28.450424 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu pid 9360
I20260619 16:44:28.461054 9670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:44:28.451302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53910 (local address 127.0.97.2:39025)
0619 16:44:28.451850 (+ 548us) server_negotiation.cc:206] Beginning negotiation
0619 16:44:28.451857 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:44:28.451885 (+ 28us) server_negotiation.cc:407] Connection header received
0619 16:44:28.451938 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:44:28.451944 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:44:28.452156 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:44:28.452299 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:44:28.453359 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:44:28.454451 (+ 1092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:44:28.455622 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:44:28.456168 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:44:28.456260 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:44:28.458468 (+ 2208us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:44:28.458476 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:44:28.458489 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:44:28.458521 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:44:28.460564 (+ 2043us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.97.2@KRBTEST.COM not found in keytab (ticket kvno 2)
0619 16:44:28.460808 (+ 244us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:53910: Request ticket server kudu/127.0.97.2@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":410,"thread_start_us":265,"threads_started":1}
W20260619 16:44:28.461144 9670 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:53910: Request ticket server kudu/127.0.97.2@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260619 16:44:28.461169 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu pid 9360: Not authorized: Client connection negotiation failed: client connection to 127.0.97.2:39025: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.97.2@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260619 16:44:28.461329 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu with pid 9360
I20260619 16:44:28.473820 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu pid 9491
I20260619 16:44:28.484920 9671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:44:28.474604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48180 (local address 127.0.97.3:45155)
0619 16:44:28.475207 (+ 603us) server_negotiation.cc:206] Beginning negotiation
0619 16:44:28.475214 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:44:28.475238 (+ 24us) server_negotiation.cc:407] Connection header received
0619 16:44:28.475291 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:44:28.475297 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:44:28.475511 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:44:28.475663 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:44:28.476797 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:44:28.477955 (+ 1158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:44:28.479200 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:44:28.479728 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:44:28.479824 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:44:28.482221 (+ 2397us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:44:28.482230 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:44:28.482243 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:44:28.482279 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:44:28.484427 (+ 2148us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.97.3@KRBTEST.COM not found in keytab (ticket kvno 2)
0619 16:44:28.484692 (+ 265us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:48180: Request ticket server kudu/127.0.97.3@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":463,"thread_start_us":308,"threads_started":1}
W20260619 16:44:28.485020 9671 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:48180: Request ticket server kudu/127.0.97.3@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260619 16:44:28.485083 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu pid 9491: Not authorized: Client connection negotiation failed: client connection to 127.0.97.3:45155: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.97.3@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260619 16:44:28.485373 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu with pid 9491
I20260619 16:44:28.497853 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu pid 9139
I20260619 16:44:28.498844 9201 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:44:28.608830 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu with pid 9139
2026-06-19T16:44:28Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (20583 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-06-19T16:44:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T16:44:28Z Disabled control of system clock
I20260619 16:44:28.674567 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu
/tmp/dist-test-taskEs2EH8/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781887385903558-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781887385903558-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781887385903558-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781887385903558-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:40779
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:45429
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:40779
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260619 16:44:28.974516 9676 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:44:28.974799 9676 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:44:28.974897 9676 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:44:28.981964 9676 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:44:28.982055 9676 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:44:28.982112 9676 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:44:28.982188 9676 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:44:28.982242 9676 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:44:28.991070 9676 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45429
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781887385903558-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781887385903558-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:40779
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:40779
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781887385903558-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781887385903558-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:44:28.992237 9676 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:44:28.993903 9676 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:44:29.004786 9684 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:44:29.004871 9682 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:44:29.005136 9681 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:44:29.005401 9676 server_base.cc:1034] running on GCE node
I20260619 16:44:29.006093 9676 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:44:29.007514 9676 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:44:29.008709 9676 hybrid_clock.cc:648] HybridClock initialized: now 1781887469008641 us; error 67 us; skew 500 ppm
I20260619 16:44:29.008988 9676 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 16:44:29.010957 9676 webserver.cc:460] Webserver started at http://127.0.97.62:41839/ using document root <none> and password file <none>
I20260619 16:44:29.011547 9676 fs_manager.cc:359] Metadata directory not provided
I20260619 16:44:29.011623 9676 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:44:29.011861 9676 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:44:29.014470 9676 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781887385903558-388-0/minicluster-data/master-0/data/instance:
uuid: "59dc3f05ed764cdaaffbc45488b1c9c6"
format_stamp: "Formatted at 2026-06-19 16:44:29 on dist-test-slave-wvz7"
server_key: "6a5c2eb71214c3fdf6d8be968f3b64d1"
server_key_iv: "a6e4a7d2be5bc1b260da377655abf896"
server_key_version: "encryptionkey@0"
I20260619 16:44:29.015126 9676 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781887385903558-388-0/minicluster-data/master-0/wal/instance:
uuid: "59dc3f05ed764cdaaffbc45488b1c9c6"
format_stamp: "Formatted at 2026-06-19 16:44:29 on dist-test-slave-wvz7"
server_key: "6a5c2eb71214c3fdf6d8be968f3b64d1"
server_key_iv: "a6e4a7d2be5bc1b260da377655abf896"
server_key_version: "encryptionkey@0"
I20260619 16:44:29.019658 9676 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 16:44:29.022782 9690 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:44:29.023944 9676 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260619 16:44:29.024062 9676 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781887385903558-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781887385903558-388-0/minicluster-data/master-0/wal
uuid: "59dc3f05ed764cdaaffbc45488b1c9c6"
format_stamp: "Formatted at 2026-06-19 16:44:29 on dist-test-slave-wvz7"
server_key: "6a5c2eb71214c3fdf6d8be968f3b64d1"
server_key_iv: "a6e4a7d2be5bc1b260da377655abf896"
server_key_version: "encryptionkey@0"
I20260619 16:44:29.024214 9676 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781887385903558-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781887385903558-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781887385903558-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:44:29.039400 9676 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:44:29.040129 9676 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:44:29.040357 9676 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:44:29.056320 9676 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:40779
I20260619 16:44:29.056340 9741 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:40779 every 8 connection(s)
I20260619 16:44:29.057545 9676 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781887385903558-388-0/minicluster-data/master-0/data/info.pb
I20260619 16:44:29.061159 9742 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:44:29.061300 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu as pid 9676
I20260619 16:44:29.061466 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781887385903558-388-0/minicluster-data/master-0/wal/instance
I20260619 16:44:29.062249 388 external_mini_cluster.cc:1428] Setting key 4076049d383ee9d7dcf294bca5114efb
I20260619 16:44:29.073289 9742 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 59dc3f05ed764cdaaffbc45488b1c9c6: Bootstrap starting.
I20260619 16:44:29.075690 9745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:44:29.064288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52286 (local address 127.0.97.62:40779)
0619 16:44:29.065266 (+ 978us) server_negotiation.cc:206] Beginning negotiation
0619 16:44:29.065278 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 16:44:29.065316 (+ 38us) server_negotiation.cc:407] Connection header received
0619 16:44:29.067422 (+ 2106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:44:29.067447 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:44:29.067970 (+ 523us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:44:29.068337 (+ 367us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:44:29.068892 (+ 555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:44:29.070131 (+ 1239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:44:29.071438 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:44:29.072114 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:44:29.072296 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:44:29.073390 (+ 1094us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:44:29.073398 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:44:29.073413 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:44:29.074020 (+ 607us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:44:29.074101 (+ 81us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:44:29.074121 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:44:29.074218 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0619 16:44:29.074445 (+ 227us) server_negotiation.cc:299] Negotiation successful
0619 16:44:29.074770 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":552,"thread_start_us":338,"threads_started":1}
I20260619 16:44:29.076727 9742 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 59dc3f05ed764cdaaffbc45488b1c9c6: Neither blocks nor log segments found. Creating new log.
I20260619 16:44:29.077834 9742 log.cc:824] T 00000000000000000000000000000000 P 59dc3f05ed764cdaaffbc45488b1c9c6: Log is configured to *not* fsync() on all Append() calls
I20260619 16:44:29.080494 9742 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 59dc3f05ed764cdaaffbc45488b1c9c6: No bootstrap required, opened a new log
I20260619 16:44:29.087476 9742 raft_consensus.cc:348] T 00000000000000000000000000000000 P 59dc3f05ed764cdaaffbc45488b1c9c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59dc3f05ed764cdaaffbc45488b1c9c6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40779 } }
I20260619 16:44:29.087826 9742 raft_consensus.cc:374] T 00000000000000000000000000000000 P 59dc3f05ed764cdaaffbc45488b1c9c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:44:29.087909 9742 raft_consensus.cc:729] T 00000000000000000000000000000000 P 59dc3f05ed764cdaaffbc45488b1c9c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59dc3f05ed764cdaaffbc45488b1c9c6, State: Initialized, Role: FOLLOWER
I20260619 16:44:29.088563 9742 consensus_queue.cc:260] T 00000000000000000000000000000000 P 59dc3f05ed764cdaaffbc45488b1c9c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59dc3f05ed764cdaaffbc45488b1c9c6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40779 } }
I20260619 16:44:29.088826 9742 raft_consensus.cc:388] T 00000000000000000000000000000000 P 59dc3f05ed764cdaaffbc45488b1c9c6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:44:29.088948 9742 raft_consensus.cc:482] T 00000000000000000000000000000000 P 59dc3f05ed764cdaaffbc45488b1c9c6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:44:29.089114 9742 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 59dc3f05ed764cdaaffbc45488b1c9c6 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:44:29.091248 9742 raft_consensus.cc:504] T 00000000000000000000000000000000 P 59dc3f05ed764cdaaffbc45488b1c9c6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59dc3f05ed764cdaaffbc45488b1c9c6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40779 } }
I20260619 16:44:29.091735 9742 leader_election.cc:302] T 00000000000000000000000000000000 P 59dc3f05ed764cdaaffbc45488b1c9c6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 59dc3f05ed764cdaaffbc45488b1c9c6; no voters:
I20260619 16:44:29.092263 9742 leader_election.cc:288] T 00000000000000000000000000000000 P 59dc3f05ed764cdaaffbc45488b1c9c6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 16:44:29.092383 9747 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 59dc3f05ed764cdaaffbc45488b1c9c6 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:44:29.093067 9747 raft_consensus.cc:686] T 00000000000000000000000000000000 P 59dc3f05ed764cdaaffbc45488b1c9c6 [term 1 LEADER]: Becoming Leader. State: Replica: 59dc3f05ed764cdaaffbc45488b1c9c6, State: Running, Role: LEADER
I20260619 16:44:29.093547 9747 consensus_queue.cc:237] T 00000000000000000000000000000000 P 59dc3f05ed764cdaaffbc45488b1c9c6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59dc3f05ed764cdaaffbc45488b1c9c6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40779 } }
I20260619 16:44:29.094231 9742 sys_catalog.cc:564] T 00000000000000000000000000000000 P 59dc3f05ed764cdaaffbc45488b1c9c6 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:44:29.096257 9748 sys_catalog.cc:455] T 00000000000000000000000000000000 P 59dc3f05ed764cdaaffbc45488b1c9c6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "59dc3f05ed764cdaaffbc45488b1c9c6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59dc3f05ed764cdaaffbc45488b1c9c6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40779 } } }
I20260619 16:44:29.096629 9748 sys_catalog.cc:458] T 00000000000000000000000000000000 P 59dc3f05ed764cdaaffbc45488b1c9c6 [sys.catalog]: This master's current role is: LEADER
I20260619 16:44:29.096254 9749 sys_catalog.cc:455] T 00000000000000000000000000000000 P 59dc3f05ed764cdaaffbc45488b1c9c6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 59dc3f05ed764cdaaffbc45488b1c9c6. Latest consensus state: current_term: 1 leader_uuid: "59dc3f05ed764cdaaffbc45488b1c9c6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59dc3f05ed764cdaaffbc45488b1c9c6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40779 } } }
I20260619 16:44:29.098129 9749 sys_catalog.cc:458] T 00000000000000000000000000000000 P 59dc3f05ed764cdaaffbc45488b1c9c6 [sys.catalog]: This master's current role is: LEADER
I20260619 16:44:29.103178 9754 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:44:29.108290 9754 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:44:29.116678 9754 catalog_manager.cc:1347] Generated new cluster ID: 66543065c3ed46258d8105fd52144345
I20260619 16:44:29.116791 9754 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:44:29.152827 9754 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 16:44:29.154141 9754 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:44:29.165455 9754 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 59dc3f05ed764cdaaffbc45488b1c9c6: Generated new TSK 0
I20260619 16:44:29.166275 9754 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 16:44:29.176270 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu
/tmp/dist-test-taskEs2EH8/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781887385903558-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781887385903558-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781887385903558-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781887385903558-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40779
--builtin_ntp_servers=127.0.97.20:45429
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260619 16:44:29.481745 9766 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:44:29.482007 9766 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:44:29.482110 9766 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:44:29.489118 9766 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:44:29.489203 9766 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:44:29.489331 9766 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260619 16:44:29.498162 9766 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45429
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781887385903558-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781887385903558-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781887385903558-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40779
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781887385903558-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:44:29.499339 9766 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:44:29.500977 9766 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:44:29.514474 9771 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:44:29.514544 9772 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:44:29.514595 9774 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:44:29.516028 9766 server_base.cc:1034] running on GCE node
I20260619 16:44:29.516708 9766 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:44:29.517977 9766 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:44:29.519248 9766 hybrid_clock.cc:648] HybridClock initialized: now 1781887469519183 us; error 57 us; skew 500 ppm
I20260619 16:44:29.519548 9766 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 16:44:29.521600 9766 webserver.cc:460] Webserver started at http://127.0.97.1:33913/ using document root <none> and password file <none>
I20260619 16:44:29.522255 9766 fs_manager.cc:359] Metadata directory not provided
I20260619 16:44:29.522341 9766 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:44:29.522631 9766 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:44:29.525338 9766 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781887385903558-388-0/minicluster-data/ts-0/data/instance:
uuid: "a626c9484dc244a299d01001d63ef8d5"
format_stamp: "Formatted at 2026-06-19 16:44:29 on dist-test-slave-wvz7"
server_key: "2d3f4313f931b57d5a44f9ce4868279f"
server_key_iv: "d7b6a3df7a14628d9f0298004e36cdd4"
server_key_version: "encryptionkey@0"
I20260619 16:44:29.526118 9766 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781887385903558-388-0/minicluster-data/ts-0/wal/instance:
uuid: "a626c9484dc244a299d01001d63ef8d5"
format_stamp: "Formatted at 2026-06-19 16:44:29 on dist-test-slave-wvz7"
server_key: "2d3f4313f931b57d5a44f9ce4868279f"
server_key_iv: "d7b6a3df7a14628d9f0298004e36cdd4"
server_key_version: "encryptionkey@0"
I20260619 16:44:29.530990 9766 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260619 16:44:29.534293 9780 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:44:29.535647 9766 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260619 16:44:29.535786 9766 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781887385903558-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781887385903558-388-0/minicluster-data/ts-0/wal
uuid: "a626c9484dc244a299d01001d63ef8d5"
format_stamp: "Formatted at 2026-06-19 16:44:29 on dist-test-slave-wvz7"
server_key: "2d3f4313f931b57d5a44f9ce4868279f"
server_key_iv: "d7b6a3df7a14628d9f0298004e36cdd4"
server_key_version: "encryptionkey@0"
I20260619 16:44:29.535977 9766 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781887385903558-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781887385903558-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781887385903558-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:44:29.582999 9766 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:44:29.583801 9766 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:44:29.584028 9766 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:44:29.584892 9766 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:44:29.586278 9766 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:44:29.586359 9766 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:44:29.586459 9766 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:44:29.586532 9766 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:44:29.614482 9766 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:37947
I20260619 16:44:29.614524 9892 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:37947 every 8 connection(s)
I20260619 16:44:29.615710 9766 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781887385903558-388-0/minicluster-data/ts-0/data/info.pb
I20260619 16:44:29.626004 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu as pid 9766
I20260619 16:44:29.626199 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781887385903558-388-0/minicluster-data/ts-0/wal/instance
I20260619 16:44:29.626940 388 external_mini_cluster.cc:1428] Setting key 07156939d31b9f57706ed3e462420db5
I20260619 16:44:29.627326 9895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:44:29.618053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:47997 (local address 127.0.97.62:40779)
0619 16:44:29.618437 (+ 384us) server_negotiation.cc:206] Beginning negotiation
0619 16:44:29.618442 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:44:29.619612 (+ 1170us) server_negotiation.cc:407] Connection header received
0619 16:44:29.621300 (+ 1688us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:44:29.621305 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:44:29.621382 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:44:29.621473 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:44:29.622507 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:44:29.623229 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:44:29.624685 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:44:29.625062 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:44:29.625166 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:44:29.626536 (+ 1370us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:44:29.626541 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:44:29.626544 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:44:29.626560 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:44:29.626602 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:44:29.626615 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:44:29.626724 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0619 16:44:29.626940 (+ 216us) server_negotiation.cc:299] Negotiation successful
0619 16:44:29.627021 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":282,"thread_start_us":188,"threads_started":1}
I20260619 16:44:29.628396 9896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:44:29.618655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40779 (local address 127.0.97.1:47997)
0619 16:44:29.619466 (+ 811us) negotiation.cc:107] Waiting for socket to connect
0619 16:44:29.619507 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0619 16:44:29.621058 (+ 1551us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:44:29.621702 (+ 644us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:44:29.621710 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:44:29.621823 (+ 113us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:44:29.622360 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:44:29.622376 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:44:29.623355 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:44:29.623358 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:44:29.624555 (+ 1197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:44:29.624571 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:44:29.625252 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:44:29.625260 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:44:29.625474 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:44:29.626208 (+ 734us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 16:44:29.626211 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:44:29.626262 (+ 51us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 16:44:29.626353 (+ 91us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:44:29.626742 (+ 389us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:44:29.626752 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:44:29.626760 (+ 8us) client_negotiation.cc:769] Sending connection context
0619 16:44:29.626926 (+ 166us) client_negotiation.cc:240] Negotiation successful
0619 16:44:29.627365 (+ 439us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":393,"thread_start_us":232,"threads_started":1}
I20260619 16:44:29.629439 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu
/tmp/dist-test-taskEs2EH8/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781887385903558-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781887385903558-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781887385903558-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781887385903558-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40779
--builtin_ntp_servers=127.0.97.20:45429
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260619 16:44:29.629807 9893 heartbeater.cc:347] Connected to a master server at 127.0.97.62:40779
I20260619 16:44:29.630086 9893 heartbeater.cc:464] Registering TS with master...
I20260619 16:44:29.630905 9893 heartbeater.cc:511] Master 127.0.97.62:40779 requested a full tablet report, sending...
I20260619 16:44:29.632764 9707 ts_manager.cc:194] Registered new tserver with Master: a626c9484dc244a299d01001d63ef8d5 (127.0.97.1:37947)
I20260619 16:44:29.634392 9707 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.1:47997
W20260619 16:44:29.932833 9897 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:44:29.933100 9897 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:44:29.933223 9897 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:44:29.940325 9897 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:44:29.940420 9897 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:44:29.940562 9897 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260619 16:44:29.949250 9897 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45429
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781887385903558-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781887385903558-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781887385903558-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40779
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781887385903558-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:44:29.950459 9897 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:44:29.952109 9897 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:44:29.963846 9902 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:44:29.963898 9903 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:44:29.963922 9905 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:44:29.964674 9897 server_base.cc:1034] running on GCE node
I20260619 16:44:29.965286 9897 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:44:29.966274 9897 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:44:29.967468 9897 hybrid_clock.cc:648] HybridClock initialized: now 1781887469967410 us; error 54 us; skew 500 ppm
I20260619 16:44:29.967770 9897 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 16:44:29.969960 9897 webserver.cc:460] Webserver started at http://127.0.97.2:33759/ using document root <none> and password file <none>
I20260619 16:44:29.970619 9897 fs_manager.cc:359] Metadata directory not provided
I20260619 16:44:29.970718 9897 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:44:29.971016 9897 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:44:29.973677 9897 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781887385903558-388-0/minicluster-data/ts-1/data/instance:
uuid: "a852b6f319da47dfaba5e8cbe73e460a"
format_stamp: "Formatted at 2026-06-19 16:44:29 on dist-test-slave-wvz7"
server_key: "a878f5dcb889c07607fcdcab032725e5"
server_key_iv: "0a93f4f009480fa844f6935b2163ba9d"
server_key_version: "encryptionkey@0"
I20260619 16:44:29.974472 9897 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781887385903558-388-0/minicluster-data/ts-1/wal/instance:
uuid: "a852b6f319da47dfaba5e8cbe73e460a"
format_stamp: "Formatted at 2026-06-19 16:44:29 on dist-test-slave-wvz7"
server_key: "a878f5dcb889c07607fcdcab032725e5"
server_key_iv: "0a93f4f009480fa844f6935b2163ba9d"
server_key_version: "encryptionkey@0"
I20260619 16:44:29.979310 9897 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260619 16:44:29.982528 9911 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:44:29.983911 9897 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260619 16:44:29.984052 9897 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781887385903558-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781887385903558-388-0/minicluster-data/ts-1/wal
uuid: "a852b6f319da47dfaba5e8cbe73e460a"
format_stamp: "Formatted at 2026-06-19 16:44:29 on dist-test-slave-wvz7"
server_key: "a878f5dcb889c07607fcdcab032725e5"
server_key_iv: "0a93f4f009480fa844f6935b2163ba9d"
server_key_version: "encryptionkey@0"
I20260619 16:44:29.984239 9897 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781887385903558-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781887385903558-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781887385903558-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:44:29.995734 9897 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:44:29.996519 9897 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:44:29.996762 9897 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:44:29.997673 9897 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:44:29.999148 9897 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:44:29.999246 9897 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:44:29.999351 9897 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:44:29.999429 9897 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:44:30.028637 9897 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:37605
I20260619 16:44:30.028684 10023 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:37605 every 8 connection(s)
I20260619 16:44:30.029975 9897 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781887385903558-388-0/minicluster-data/ts-1/data/info.pb
I20260619 16:44:30.038450 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu as pid 9897
I20260619 16:44:30.038615 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781887385903558-388-0/minicluster-data/ts-1/wal/instance
I20260619 16:44:30.039086 388 external_mini_cluster.cc:1428] Setting key 8252dff692a3ea5c2dd6f681290d0fcf
I20260619 16:44:30.041651 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu
/tmp/dist-test-taskEs2EH8/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781887385903558-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781887385903558-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781887385903558-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781887385903558-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40779
--builtin_ntp_servers=127.0.97.20:45429
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260619 16:44:30.042994 9895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:44:30.032394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:33995 (local address 127.0.97.62:40779)
0619 16:44:30.032548 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0619 16:44:30.032552 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:44:30.033990 (+ 1438us) server_negotiation.cc:407] Connection header received
0619 16:44:30.035721 (+ 1731us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:44:30.035727 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:44:30.035799 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:44:30.035885 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:44:30.036909 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:44:30.037619 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:44:30.039346 (+ 1727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:44:30.039810 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:44:30.039955 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:44:30.042278 (+ 2323us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:44:30.042283 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:44:30.042286 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:44:30.042304 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:44:30.042345 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:44:30.042357 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:44:30.042420 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0619 16:44:30.042701 (+ 281us) server_negotiation.cc:299] Negotiation successful
0619 16:44:30.042777 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260619 16:44:30.044302 10026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:44:30.032894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40779 (local address 127.0.97.2:33995)
0619 16:44:30.033846 (+ 952us) negotiation.cc:107] Waiting for socket to connect
0619 16:44:30.033888 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0619 16:44:30.035413 (+ 1525us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:44:30.036079 (+ 666us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:44:30.036105 (+ 26us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:44:30.036200 (+ 95us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:44:30.036743 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:44:30.036759 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:44:30.037788 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:44:30.037795 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:44:30.039213 (+ 1418us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:44:30.039228 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:44:30.040085 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:44:30.040092 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:44:30.040395 (+ 303us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:44:30.041931 (+ 1536us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 16:44:30.041934 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:44:30.041988 (+ 54us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 16:44:30.042078 (+ 90us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:44:30.042438 (+ 360us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:44:30.042447 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:44:30.042454 (+ 7us) client_negotiation.cc:769] Sending connection context
0619 16:44:30.042628 (+ 174us) client_negotiation.cc:240] Negotiation successful
0619 16:44:30.043042 (+ 414us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":462,"thread_start_us":287,"threads_started":1}
I20260619 16:44:30.045644 10024 heartbeater.cc:347] Connected to a master server at 127.0.97.62:40779
I20260619 16:44:30.045962 10024 heartbeater.cc:464] Registering TS with master...
I20260619 16:44:30.046710 10024 heartbeater.cc:511] Master 127.0.97.62:40779 requested a full tablet report, sending...
I20260619 16:44:30.047968 9707 ts_manager.cc:194] Registered new tserver with Master: a852b6f319da47dfaba5e8cbe73e460a (127.0.97.2:37605)
I20260619 16:44:30.048868 9707 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.2:33995
W20260619 16:44:30.352816 10027 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:44:30.353072 10027 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:44:30.353163 10027 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:44:30.360038 10027 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:44:30.360129 10027 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:44:30.360227 10027 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260619 16:44:30.368856 10027 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45429
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781887385903558-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781887385903558-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781887385903558-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40779
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781887385903558-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:44:30.369999 10027 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:44:30.371654 10027 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:44:30.383344 10035 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:44:30.383471 10032 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:44:30.383361 10033 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:44:30.384104 10027 server_base.cc:1034] running on GCE node
I20260619 16:44:30.384732 10027 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:44:30.385560 10027 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:44:30.386778 10027 hybrid_clock.cc:648] HybridClock initialized: now 1781887470386717 us; error 61 us; skew 500 ppm
I20260619 16:44:30.387060 10027 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 16:44:30.389061 10027 webserver.cc:460] Webserver started at http://127.0.97.3:35225/ using document root <none> and password file <none>
I20260619 16:44:30.389668 10027 fs_manager.cc:359] Metadata directory not provided
I20260619 16:44:30.389781 10027 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:44:30.390061 10027 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:44:30.392671 10027 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781887385903558-388-0/minicluster-data/ts-2/data/instance:
uuid: "b851595e9a244eb1b72bdea0b22b43fd"
format_stamp: "Formatted at 2026-06-19 16:44:30 on dist-test-slave-wvz7"
server_key: "0b027e593069350747bce00128ec021f"
server_key_iv: "626e2fb5215da8d3226cbdaca4f24fd7"
server_key_version: "encryptionkey@0"
I20260619 16:44:30.393379 10027 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781887385903558-388-0/minicluster-data/ts-2/wal/instance:
uuid: "b851595e9a244eb1b72bdea0b22b43fd"
format_stamp: "Formatted at 2026-06-19 16:44:30 on dist-test-slave-wvz7"
server_key: "0b027e593069350747bce00128ec021f"
server_key_iv: "626e2fb5215da8d3226cbdaca4f24fd7"
server_key_version: "encryptionkey@0"
I20260619 16:44:30.398123 10027 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260619 16:44:30.401222 10041 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:44:30.402590 10027 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 16:44:30.402724 10027 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781887385903558-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781887385903558-388-0/minicluster-data/ts-2/wal
uuid: "b851595e9a244eb1b72bdea0b22b43fd"
format_stamp: "Formatted at 2026-06-19 16:44:30 on dist-test-slave-wvz7"
server_key: "0b027e593069350747bce00128ec021f"
server_key_iv: "626e2fb5215da8d3226cbdaca4f24fd7"
server_key_version: "encryptionkey@0"
I20260619 16:44:30.402897 10027 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781887385903558-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781887385903558-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781887385903558-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:44:30.433938 10027 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:44:30.434763 10027 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:44:30.435002 10027 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:44:30.435909 10027 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:44:30.437384 10027 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:44:30.437476 10027 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:44:30.437582 10027 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:44:30.437649 10027 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:44:30.466353 10027 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:33507
I20260619 16:44:30.466427 10153 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:33507 every 8 connection(s)
I20260619 16:44:30.467574 10027 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781887385903558-388-0/minicluster-data/ts-2/data/info.pb
I20260619 16:44:30.472878 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu as pid 10027
I20260619 16:44:30.473009 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781887385903558-388-0/minicluster-data/ts-2/wal/instance
I20260619 16:44:30.473498 388 external_mini_cluster.cc:1428] Setting key 212854731a431f2d6d96ca2b02c62835
I20260619 16:44:30.479475 9895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:44:30.469996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:49103 (local address 127.0.97.62:40779)
0619 16:44:30.470201 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0619 16:44:30.470207 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:44:30.471553 (+ 1346us) server_negotiation.cc:407] Connection header received
0619 16:44:30.473245 (+ 1692us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:44:30.473251 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:44:30.473333 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:44:30.473430 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:44:30.474563 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:44:30.475307 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:44:30.476780 (+ 1473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:44:30.477193 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:44:30.477298 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:44:30.478599 (+ 1301us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:44:30.478606 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:44:30.478612 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:44:30.478634 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:44:30.478665 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:44:30.478679 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:44:30.478763 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0619 16:44:30.478977 (+ 214us) server_negotiation.cc:299] Negotiation successful
0619 16:44:30.479095 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260619 16:44:30.480419 10156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:44:30.470554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40779 (local address 127.0.97.3:49103)
0619 16:44:30.471410 (+ 856us) negotiation.cc:107] Waiting for socket to connect
0619 16:44:30.471450 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0619 16:44:30.472949 (+ 1499us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:44:30.473608 (+ 659us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:44:30.473619 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:44:30.473773 (+ 154us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:44:30.474409 (+ 636us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:44:30.474426 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:44:30.475441 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:44:30.475447 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:44:30.476655 (+ 1208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:44:30.476666 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:44:30.477386 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:44:30.477392 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:44:30.477600 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:44:30.478282 (+ 682us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 16:44:30.478285 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:44:30.478324 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 16:44:30.478398 (+ 74us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:44:30.478774 (+ 376us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:44:30.478785 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:44:30.478794 (+ 9us) client_negotiation.cc:769] Sending connection context
0619 16:44:30.478956 (+ 162us) client_negotiation.cc:240] Negotiation successful
0619 16:44:30.479333 (+ 377us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":423,"thread_start_us":260,"threads_started":1}
I20260619 16:44:30.481686 10154 heartbeater.cc:347] Connected to a master server at 127.0.97.62:40779
I20260619 16:44:30.482005 10154 heartbeater.cc:464] Registering TS with master...
I20260619 16:44:30.482719 10154 heartbeater.cc:511] Master 127.0.97.62:40779 requested a full tablet report, sending...
I20260619 16:44:30.483933 9707 ts_manager.cc:194] Registered new tserver with Master: b851595e9a244eb1b72bdea0b22b43fd (127.0.97.3:33507)
I20260619 16:44:30.484733 9707 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.3:49103
I20260619 16:44:30.487922 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260619 16:44:30.493299 9895 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:44:30.491948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52296 (local address 127.0.97.62:40779)
0619 16:44:30.492089 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0619 16:44:30.492094 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:44:30.492505 (+ 411us) server_negotiation.cc:407] Connection header received
0619 16:44:30.492626 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:44:30.492630 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:44:30.492697 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:44:30.492769 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:44:30.493102 (+ 333us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52296: BlockingRecv error: recv got EOF from 127.0.0.1:52296 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
I20260619 16:44:30.494009 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu pid 9766
I20260619 16:44:30.501616 10163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:44:30.494915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37226 (local address 127.0.97.1:37947)
0619 16:44:30.495722 (+ 807us) server_negotiation.cc:206] Beginning negotiation
0619 16:44:30.495729 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:44:30.495751 (+ 22us) server_negotiation.cc:407] Connection header received
0619 16:44:30.495799 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:44:30.495804 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:44:30.495986 (+ 182us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:44:30.496125 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:44:30.496513 (+ 388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:44:30.497467 (+ 954us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:44:30.498664 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:44:30.499192 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:44:30.499303 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:44:30.500258 (+ 955us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:44:30.500266 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:44:30.500277 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:44:30.500870 (+ 593us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:44:30.500915 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:44:30.500933 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:44:30.501029 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0619 16:44:30.501174 (+ 145us) server_negotiation.cc:299] Negotiation successful
0619 16:44:30.501306 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":683,"thread_start_us":556,"threads_started":1}
I20260619 16:44:30.502816 9888 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:44:30.583253 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu with pid 9766
I20260619 16:44:30.595772 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu pid 9897
I20260619 16:44:30.603199 10165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:44:30.596578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33956 (local address 127.0.97.2:37605)
0619 16:44:30.597059 (+ 481us) server_negotiation.cc:206] Beginning negotiation
0619 16:44:30.597066 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:44:30.597092 (+ 26us) server_negotiation.cc:407] Connection header received
0619 16:44:30.597153 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:44:30.597159 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:44:30.597389 (+ 230us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:44:30.597512 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:44:30.597953 (+ 441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:44:30.598944 (+ 991us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:44:30.600080 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:44:30.600661 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:44:30.600766 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:44:30.601666 (+ 900us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:44:30.601674 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:44:30.601688 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:44:30.602438 (+ 750us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:44:30.602482 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:44:30.602500 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:44:30.602592 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0619 16:44:30.602715 (+ 123us) server_negotiation.cc:299] Negotiation successful
0619 16:44:30.602855 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":352,"thread_start_us":249,"threads_started":1}
I20260619 16:44:30.604388 10019 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:44:30.682935 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu with pid 9897
I20260619 16:44:30.694813 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu pid 10027
I20260619 16:44:30.702337 10167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:44:30.695754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55784 (local address 127.0.97.3:33507)
0619 16:44:30.696326 (+ 572us) server_negotiation.cc:206] Beginning negotiation
0619 16:44:30.696333 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:44:30.696360 (+ 27us) server_negotiation.cc:407] Connection header received
0619 16:44:30.696416 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:44:30.696421 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:44:30.696656 (+ 235us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:44:30.696781 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:44:30.697136 (+ 355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:44:30.698198 (+ 1062us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:44:30.699380 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:44:30.699911 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:44:30.700001 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:44:30.700946 (+ 945us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:44:30.700954 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:44:30.700967 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:44:30.701519 (+ 552us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:44:30.701602 (+ 83us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:44:30.701623 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:44:30.701709 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0619 16:44:30.701865 (+ 156us) server_negotiation.cc:299] Negotiation successful
0619 16:44:30.701976 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":442,"thread_start_us":293,"threads_started":1}
I20260619 16:44:30.703552 10149 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:44:30.782713 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu with pid 10027
I20260619 16:44:30.794360 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu pid 9676
I20260619 16:44:30.795457 9737 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:44:30.891778 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu with pid 9676
2026-06-19T16:44:30Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (2278 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-06-19T16:44:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T16:44:30Z Disabled control of system clock
I20260619 16:44:30.950618 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu
/tmp/dist-test-taskEs2EH8/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781887385903558-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781887385903558-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781887385903558-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781887385903558-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:36913
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:44565
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:36913
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260619 16:44:31.249953 10173 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:44:31.250228 10173 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:44:31.250315 10173 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:44:31.257390 10173 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:44:31.257500 10173 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:44:31.257535 10173 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:44:31.257591 10173 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:44:31.257639 10173 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:44:31.266768 10173 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44565
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781887385903558-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781887385903558-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:36913
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:36913
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781887385903558-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781887385903558-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:44:31.267963 10173 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:44:31.269586 10173 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:44:31.280745 10178 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:44:31.280673 10181 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:44:31.280755 10179 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:44:31.281316 10173 server_base.cc:1034] running on GCE node
I20260619 16:44:31.282115 10173 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:44:31.283486 10173 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:44:31.284713 10173 hybrid_clock.cc:648] HybridClock initialized: now 1781887471284658 us; error 41 us; skew 500 ppm
I20260619 16:44:31.285039 10173 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 16:44:31.287213 10173 webserver.cc:460] Webserver started at http://127.0.97.62:43933/ using document root <none> and password file <none>
I20260619 16:44:31.287853 10173 fs_manager.cc:359] Metadata directory not provided
I20260619 16:44:31.287947 10173 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:44:31.288236 10173 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:44:31.290939 10173 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781887385903558-388-0/minicluster-data/master-0/data/instance:
uuid: "44ee93a7cebb451bb6bb6119d749aebc"
format_stamp: "Formatted at 2026-06-19 16:44:31 on dist-test-slave-wvz7"
server_key: "4c6a20f0e19631474767a80c21b73852"
server_key_iv: "2730c7fcdff251fc805ec9fda858b415"
server_key_version: "encryptionkey@0"
I20260619 16:44:31.291698 10173 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781887385903558-388-0/minicluster-data/master-0/wal/instance:
uuid: "44ee93a7cebb451bb6bb6119d749aebc"
format_stamp: "Formatted at 2026-06-19 16:44:31 on dist-test-slave-wvz7"
server_key: "4c6a20f0e19631474767a80c21b73852"
server_key_iv: "2730c7fcdff251fc805ec9fda858b415"
server_key_version: "encryptionkey@0"
I20260619 16:44:31.296509 10173 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 16:44:31.299651 10187 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:44:31.300859 10173 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 16:44:31.301000 10173 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781887385903558-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781887385903558-388-0/minicluster-data/master-0/wal
uuid: "44ee93a7cebb451bb6bb6119d749aebc"
format_stamp: "Formatted at 2026-06-19 16:44:31 on dist-test-slave-wvz7"
server_key: "4c6a20f0e19631474767a80c21b73852"
server_key_iv: "2730c7fcdff251fc805ec9fda858b415"
server_key_version: "encryptionkey@0"
I20260619 16:44:31.301182 10173 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781887385903558-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781887385903558-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781887385903558-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:44:31.306319 10173 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:44:31.306964 10173 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:44:31.307209 10173 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:44:31.323108 10173 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:36913
I20260619 16:44:31.323138 10238 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:36913 every 8 connection(s)
I20260619 16:44:31.324249 10173 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781887385903558-388-0/minicluster-data/master-0/data/info.pb
I20260619 16:44:31.326306 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu as pid 10173
I20260619 16:44:31.326493 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781887385903558-388-0/minicluster-data/master-0/wal/instance
I20260619 16:44:31.327117 388 external_mini_cluster.cc:1428] Setting key 66400adacbbc1b6d6d4d82260b9d1278
I20260619 16:44:31.327770 10239 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:44:31.336146 10242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:44:31.328973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49040 (local address 127.0.97.62:36913)
0619 16:44:31.330055 (+ 1082us) server_negotiation.cc:206] Beginning negotiation
0619 16:44:31.330070 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 16:44:31.330117 (+ 47us) server_negotiation.cc:407] Connection header received
0619 16:44:31.332204 (+ 2087us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:44:31.332231 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:44:31.332783 (+ 552us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:44:31.333223 (+ 440us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:44:31.334229 (+ 1006us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:44:31.334237 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:44:31.334269 (+ 32us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:44:31.334289 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:44:31.334366 (+ 77us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:44:31.334388 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:44:31.334603 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0619 16:44:31.334743 (+ 140us) server_negotiation.cc:299] Negotiation successful
0619 16:44:31.335015 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":561,"thread_start_us":344,"threads_started":1}
I20260619 16:44:31.340802 10239 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 44ee93a7cebb451bb6bb6119d749aebc: Bootstrap starting.
I20260619 16:44:31.343637 10239 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 44ee93a7cebb451bb6bb6119d749aebc: Neither blocks nor log segments found. Creating new log.
I20260619 16:44:31.344537 10239 log.cc:824] T 00000000000000000000000000000000 P 44ee93a7cebb451bb6bb6119d749aebc: Log is configured to *not* fsync() on all Append() calls
I20260619 16:44:31.347038 10239 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 44ee93a7cebb451bb6bb6119d749aebc: No bootstrap required, opened a new log
I20260619 16:44:31.353631 10239 raft_consensus.cc:348] T 00000000000000000000000000000000 P 44ee93a7cebb451bb6bb6119d749aebc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44ee93a7cebb451bb6bb6119d749aebc" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36913 } }
I20260619 16:44:31.353967 10239 raft_consensus.cc:374] T 00000000000000000000000000000000 P 44ee93a7cebb451bb6bb6119d749aebc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:44:31.354054 10239 raft_consensus.cc:729] T 00000000000000000000000000000000 P 44ee93a7cebb451bb6bb6119d749aebc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44ee93a7cebb451bb6bb6119d749aebc, State: Initialized, Role: FOLLOWER
I20260619 16:44:31.354552 10239 consensus_queue.cc:260] T 00000000000000000000000000000000 P 44ee93a7cebb451bb6bb6119d749aebc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44ee93a7cebb451bb6bb6119d749aebc" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36913 } }
I20260619 16:44:31.354734 10239 raft_consensus.cc:388] T 00000000000000000000000000000000 P 44ee93a7cebb451bb6bb6119d749aebc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:44:31.354844 10239 raft_consensus.cc:482] T 00000000000000000000000000000000 P 44ee93a7cebb451bb6bb6119d749aebc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:44:31.354974 10239 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 44ee93a7cebb451bb6bb6119d749aebc [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:44:31.357017 10239 raft_consensus.cc:504] T 00000000000000000000000000000000 P 44ee93a7cebb451bb6bb6119d749aebc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44ee93a7cebb451bb6bb6119d749aebc" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36913 } }
I20260619 16:44:31.357450 10239 leader_election.cc:302] T 00000000000000000000000000000000 P 44ee93a7cebb451bb6bb6119d749aebc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 44ee93a7cebb451bb6bb6119d749aebc; no voters:
I20260619 16:44:31.357975 10239 leader_election.cc:288] T 00000000000000000000000000000000 P 44ee93a7cebb451bb6bb6119d749aebc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 16:44:31.358083 10244 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 44ee93a7cebb451bb6bb6119d749aebc [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:44:31.358709 10244 raft_consensus.cc:686] T 00000000000000000000000000000000 P 44ee93a7cebb451bb6bb6119d749aebc [term 1 LEADER]: Becoming Leader. State: Replica: 44ee93a7cebb451bb6bb6119d749aebc, State: Running, Role: LEADER
I20260619 16:44:31.359187 10244 consensus_queue.cc:237] T 00000000000000000000000000000000 P 44ee93a7cebb451bb6bb6119d749aebc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44ee93a7cebb451bb6bb6119d749aebc" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36913 } }
I20260619 16:44:31.359752 10239 sys_catalog.cc:564] T 00000000000000000000000000000000 P 44ee93a7cebb451bb6bb6119d749aebc [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:44:31.361575 10246 sys_catalog.cc:455] T 00000000000000000000000000000000 P 44ee93a7cebb451bb6bb6119d749aebc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 44ee93a7cebb451bb6bb6119d749aebc. Latest consensus state: current_term: 1 leader_uuid: "44ee93a7cebb451bb6bb6119d749aebc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44ee93a7cebb451bb6bb6119d749aebc" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36913 } } }
I20260619 16:44:31.361986 10246 sys_catalog.cc:458] T 00000000000000000000000000000000 P 44ee93a7cebb451bb6bb6119d749aebc [sys.catalog]: This master's current role is: LEADER
I20260619 16:44:31.362098 10245 sys_catalog.cc:455] T 00000000000000000000000000000000 P 44ee93a7cebb451bb6bb6119d749aebc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "44ee93a7cebb451bb6bb6119d749aebc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44ee93a7cebb451bb6bb6119d749aebc" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36913 } } }
I20260619 16:44:31.362394 10245 sys_catalog.cc:458] T 00000000000000000000000000000000 P 44ee93a7cebb451bb6bb6119d749aebc [sys.catalog]: This master's current role is: LEADER
I20260619 16:44:31.363704 10253 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:44:31.368422 10253 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:44:31.377463 10253 catalog_manager.cc:1347] Generated new cluster ID: 83d170863f92444eae503908e9535786
I20260619 16:44:31.377638 10253 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:44:31.393911 10253 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 16:44:31.394309 10253 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:44:31.410941 10253 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 44ee93a7cebb451bb6bb6119d749aebc: Generated new TSK 0
I20260619 16:44:31.411751 10253 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 16:44:31.422801 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu
/tmp/dist-test-taskEs2EH8/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781887385903558-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781887385903558-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781887385903558-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781887385903558-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36913
--builtin_ntp_servers=127.0.97.20:44565
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260619 16:44:31.726768 10263 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:44:31.727010 10263 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:44:31.727101 10263 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:44:31.733992 10263 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:44:31.734088 10263 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:44:31.734191 10263 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260619 16:44:31.742970 10263 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44565
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781887385903558-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781887385903558-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781887385903558-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36913
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781887385903558-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:44:31.744136 10263 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:44:31.745757 10263 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:44:31.757360 10268 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:44:31.757313 10271 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:44:31.757397 10269 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:44:31.758049 10263 server_base.cc:1034] running on GCE node
I20260619 16:44:31.758559 10263 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:44:31.759411 10263 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:44:31.760593 10263 hybrid_clock.cc:648] HybridClock initialized: now 1781887471760539 us; error 48 us; skew 500 ppm
I20260619 16:44:31.760902 10263 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 16:44:31.762907 10263 webserver.cc:460] Webserver started at http://127.0.97.1:32915/ using document root <none> and password file <none>
I20260619 16:44:31.763527 10263 fs_manager.cc:359] Metadata directory not provided
I20260619 16:44:31.763609 10263 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:44:31.763852 10263 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:44:31.766408 10263 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781887385903558-388-0/minicluster-data/ts-0/data/instance:
uuid: "a0cebd3f4c8244bd8aabb2302fc288f1"
format_stamp: "Formatted at 2026-06-19 16:44:31 on dist-test-slave-wvz7"
server_key: "4c6f5135b428515d0e7c3b613100e692"
server_key_iv: "4d4d670ebd8cbd39dbc641d6f854c20c"
server_key_version: "encryptionkey@0"
I20260619 16:44:31.767074 10263 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781887385903558-388-0/minicluster-data/ts-0/wal/instance:
uuid: "a0cebd3f4c8244bd8aabb2302fc288f1"
format_stamp: "Formatted at 2026-06-19 16:44:31 on dist-test-slave-wvz7"
server_key: "4c6f5135b428515d0e7c3b613100e692"
server_key_iv: "4d4d670ebd8cbd39dbc641d6f854c20c"
server_key_version: "encryptionkey@0"
I20260619 16:44:31.771553 10263 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 16:44:31.774539 10277 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:44:31.775710 10263 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 16:44:31.775851 10263 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781887385903558-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781887385903558-388-0/minicluster-data/ts-0/wal
uuid: "a0cebd3f4c8244bd8aabb2302fc288f1"
format_stamp: "Formatted at 2026-06-19 16:44:31 on dist-test-slave-wvz7"
server_key: "4c6f5135b428515d0e7c3b613100e692"
server_key_iv: "4d4d670ebd8cbd39dbc641d6f854c20c"
server_key_version: "encryptionkey@0"
I20260619 16:44:31.776036 10263 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781887385903558-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781887385903558-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781887385903558-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:44:31.780988 10263 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:44:31.781644 10263 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:44:31.781893 10263 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:44:31.782686 10263 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:44:31.783872 10263 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:44:31.783960 10263 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:44:31.784056 10263 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:44:31.784122 10263 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:44:31.811240 10263 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:38129
I20260619 16:44:31.811299 10389 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:38129 every 8 connection(s)
I20260619 16:44:31.812446 10263 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781887385903558-388-0/minicluster-data/ts-0/data/info.pb
I20260619 16:44:31.819736 10242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:44:31.814757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:40079 (local address 127.0.97.62:36913)
0619 16:44:31.814910 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0619 16:44:31.814914 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:44:31.816148 (+ 1234us) server_negotiation.cc:407] Connection header received
0619 16:44:31.817780 (+ 1632us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:44:31.817786 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:44:31.817846 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:44:31.817931 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:44:31.819051 (+ 1120us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:44:31.819057 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:44:31.819061 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:44:31.819076 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:44:31.819105 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:44:31.819116 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:44:31.819211 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0619 16:44:31.819461 (+ 250us) server_negotiation.cc:299] Negotiation successful
0619 16:44:31.819525 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260619 16:44:31.820569 10392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:44:31.815195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36913 (local address 127.0.97.1:40079)
0619 16:44:31.816000 (+ 805us) negotiation.cc:107] Waiting for socket to connect
0619 16:44:31.816041 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0619 16:44:31.817498 (+ 1457us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:44:31.818142 (+ 644us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:44:31.818149 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:44:31.818235 (+ 86us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:44:31.818779 (+ 544us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 16:44:31.818782 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:44:31.818815 (+ 33us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 16:44:31.818892 (+ 77us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:44:31.819196 (+ 304us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:44:31.819205 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:44:31.819212 (+ 7us) client_negotiation.cc:769] Sending connection context
0619 16:44:31.819495 (+ 283us) client_negotiation.cc:240] Negotiation successful
0619 16:44:31.819718 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":398,"thread_start_us":240,"threads_started":1}
I20260619 16:44:31.821905 10390 heartbeater.cc:347] Connected to a master server at 127.0.97.62:36913
I20260619 16:44:31.821998 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu as pid 10263
I20260619 16:44:31.822154 10390 heartbeater.cc:464] Registering TS with master...
I20260619 16:44:31.822193 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781887385903558-388-0/minicluster-data/ts-0/wal/instance
I20260619 16:44:31.822685 10390 heartbeater.cc:511] Master 127.0.97.62:36913 requested a full tablet report, sending...
I20260619 16:44:31.822737 388 external_mini_cluster.cc:1428] Setting key 66457b1f9e027b772456114b1b2accb8
I20260619 16:44:31.824561 10203 ts_manager.cc:194] Registered new tserver with Master: a0cebd3f4c8244bd8aabb2302fc288f1 (127.0.97.1:38129)
I20260619 16:44:31.825109 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu
/tmp/dist-test-taskEs2EH8/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781887385903558-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781887385903558-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781887385903558-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781887385903558-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36913
--builtin_ntp_servers=127.0.97.20:44565
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260619 16:44:32.126606 10393 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:44:32.126883 10393 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:44:32.127003 10393 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:44:32.134111 10393 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:44:32.134214 10393 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:44:32.134336 10393 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260619 16:44:32.143136 10393 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44565
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781887385903558-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781887385903558-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781887385903558-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36913
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781887385903558-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:44:32.144280 10393 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:44:32.145960 10393 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:44:32.157896 10399 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:44:32.157925 10398 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:44:32.158154 10401 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:44:32.158568 10393 server_base.cc:1034] running on GCE node
I20260619 16:44:32.159070 10393 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:44:32.159897 10393 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:44:32.161079 10393 hybrid_clock.cc:648] HybridClock initialized: now 1781887472161029 us; error 46 us; skew 500 ppm
I20260619 16:44:32.161367 10393 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 16:44:32.163466 10393 webserver.cc:460] Webserver started at http://127.0.97.2:41887/ using document root <none> and password file <none>
I20260619 16:44:32.164099 10393 fs_manager.cc:359] Metadata directory not provided
I20260619 16:44:32.164192 10393 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:44:32.164463 10393 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:44:32.168541 10393 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781887385903558-388-0/minicluster-data/ts-1/data/instance:
uuid: "b7167405d6444e08853f84c8177df0dc"
format_stamp: "Formatted at 2026-06-19 16:44:32 on dist-test-slave-wvz7"
server_key: "e22046adf957bf12444cf1bc6ca95d2c"
server_key_iv: "fb5b2d21ec1268b96f5e9d94a7f26a3b"
server_key_version: "encryptionkey@0"
I20260619 16:44:32.169286 10393 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781887385903558-388-0/minicluster-data/ts-1/wal/instance:
uuid: "b7167405d6444e08853f84c8177df0dc"
format_stamp: "Formatted at 2026-06-19 16:44:32 on dist-test-slave-wvz7"
server_key: "e22046adf957bf12444cf1bc6ca95d2c"
server_key_iv: "fb5b2d21ec1268b96f5e9d94a7f26a3b"
server_key_version: "encryptionkey@0"
I20260619 16:44:32.174081 10393 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.001s
I20260619 16:44:32.177292 10407 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:44:32.178756 10393 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 16:44:32.178908 10393 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781887385903558-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781887385903558-388-0/minicluster-data/ts-1/wal
uuid: "b7167405d6444e08853f84c8177df0dc"
format_stamp: "Formatted at 2026-06-19 16:44:32 on dist-test-slave-wvz7"
server_key: "e22046adf957bf12444cf1bc6ca95d2c"
server_key_iv: "fb5b2d21ec1268b96f5e9d94a7f26a3b"
server_key_version: "encryptionkey@0"
I20260619 16:44:32.179131 10393 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781887385903558-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781887385903558-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781887385903558-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:44:32.184672 10393 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:44:32.185386 10393 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:44:32.185616 10393 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:44:32.186712 10393 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:44:32.188025 10393 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:44:32.188123 10393 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:44:32.188231 10393 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:44:32.188300 10393 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:44:32.217490 10393 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:44751
I20260619 16:44:32.217536 10519 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:44751 every 8 connection(s)
I20260619 16:44:32.218719 10393 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781887385903558-388-0/minicluster-data/ts-1/data/info.pb
I20260619 16:44:32.226128 10242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:44:32.220998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:35035 (local address 127.0.97.62:36913)
0619 16:44:32.221167 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0619 16:44:32.221172 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:44:32.222864 (+ 1692us) server_negotiation.cc:407] Connection header received
0619 16:44:32.224380 (+ 1516us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:44:32.224384 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:44:32.224436 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:44:32.224523 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:44:32.225585 (+ 1062us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:44:32.225588 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:44:32.225591 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:44:32.225602 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:44:32.225624 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:44:32.225633 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:44:32.225715 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0619 16:44:32.225875 (+ 160us) server_negotiation.cc:299] Negotiation successful
0619 16:44:32.225921 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260619 16:44:32.226933 10522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:44:32.221544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36913 (local address 127.0.97.2:35035)
0619 16:44:32.222727 (+ 1183us) negotiation.cc:107] Waiting for socket to connect
0619 16:44:32.222768 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0619 16:44:32.224157 (+ 1389us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:44:32.224678 (+ 521us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:44:32.224686 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:44:32.224772 (+ 86us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:44:32.225345 (+ 573us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 16:44:32.225348 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:44:32.225384 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 16:44:32.225457 (+ 73us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:44:32.225693 (+ 236us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:44:32.225699 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:44:32.225704 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 16:44:32.225926 (+ 222us) client_negotiation.cc:240] Negotiation successful
0619 16:44:32.226115 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":642,"thread_start_us":386,"threads_started":1}
I20260619 16:44:32.228065 10520 heartbeater.cc:347] Connected to a master server at 127.0.97.62:36913
I20260619 16:44:32.228309 10520 heartbeater.cc:464] Registering TS with master...
I20260619 16:44:32.228739 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu as pid 10393
I20260619 16:44:32.228886 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781887385903558-388-0/minicluster-data/ts-1/wal/instance
I20260619 16:44:32.228891 10520 heartbeater.cc:511] Master 127.0.97.62:36913 requested a full tablet report, sending...
I20260619 16:44:32.229254 388 external_mini_cluster.cc:1428] Setting key c80a6c87d37d95386e66db9646837706
I20260619 16:44:32.230264 10203 ts_manager.cc:194] Registered new tserver with Master: b7167405d6444e08853f84c8177df0dc (127.0.97.2:44751)
I20260619 16:44:32.231686 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu
/tmp/dist-test-taskEs2EH8/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781887385903558-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781887385903558-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781887385903558-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781887385903558-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36913
--builtin_ntp_servers=127.0.97.20:44565
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260619 16:44:32.533310 10523 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:44:32.533571 10523 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:44:32.533667 10523 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:44:32.540731 10523 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:44:32.540827 10523 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:44:32.540942 10523 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260619 16:44:32.549630 10523 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44565
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781887385903558-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781887385903558-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781887385903558-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36913
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781887385903558-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:44:32.550853 10523 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:44:32.552428 10523 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:44:32.564770 10528 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:44:32.564841 10529 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:44:32.564854 10531 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:44:32.565845 10523 server_base.cc:1034] running on GCE node
I20260619 16:44:32.566406 10523 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:44:32.567335 10523 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:44:32.568557 10523 hybrid_clock.cc:648] HybridClock initialized: now 1781887472568495 us; error 54 us; skew 500 ppm
I20260619 16:44:32.568852 10523 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 16:44:32.571028 10523 webserver.cc:460] Webserver started at http://127.0.97.3:33933/ using document root <none> and password file <none>
I20260619 16:44:32.571696 10523 fs_manager.cc:359] Metadata directory not provided
I20260619 16:44:32.571785 10523 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:44:32.572081 10523 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:44:32.574918 10523 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781887385903558-388-0/minicluster-data/ts-2/data/instance:
uuid: "9d5c3422e2b44673a5363999a8fabdc8"
format_stamp: "Formatted at 2026-06-19 16:44:32 on dist-test-slave-wvz7"
server_key: "b88adcfb7d5f3eff59d358061c49aefc"
server_key_iv: "5830e772cbab00341048384c8af2e9f6"
server_key_version: "encryptionkey@0"
I20260619 16:44:32.575681 10523 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781887385903558-388-0/minicluster-data/ts-2/wal/instance:
uuid: "9d5c3422e2b44673a5363999a8fabdc8"
format_stamp: "Formatted at 2026-06-19 16:44:32 on dist-test-slave-wvz7"
server_key: "b88adcfb7d5f3eff59d358061c49aefc"
server_key_iv: "5830e772cbab00341048384c8af2e9f6"
server_key_version: "encryptionkey@0"
I20260619 16:44:32.580830 10523 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 16:44:32.584410 10537 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:44:32.585873 10523 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 16:44:32.586023 10523 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781887385903558-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781887385903558-388-0/minicluster-data/ts-2/wal
uuid: "9d5c3422e2b44673a5363999a8fabdc8"
format_stamp: "Formatted at 2026-06-19 16:44:32 on dist-test-slave-wvz7"
server_key: "b88adcfb7d5f3eff59d358061c49aefc"
server_key_iv: "5830e772cbab00341048384c8af2e9f6"
server_key_version: "encryptionkey@0"
I20260619 16:44:32.586248 10523 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781887385903558-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781887385903558-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781887385903558-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:44:32.591797 10523 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:44:32.592547 10523 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:44:32.592775 10523 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:44:32.593601 10523 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:44:32.594940 10523 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:44:32.595021 10523 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:44:32.595130 10523 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:44:32.595201 10523 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:44:32.622905 10523 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:45377
I20260619 16:44:32.622949 10649 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:45377 every 8 connection(s)
I20260619 16:44:32.624152 10523 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781887385903558-388-0/minicluster-data/ts-2/data/info.pb
I20260619 16:44:32.629133 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu as pid 10523
I20260619 16:44:32.629307 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781887385903558-388-0/minicluster-data/ts-2/wal/instance
I20260619 16:44:32.629690 388 external_mini_cluster.cc:1428] Setting key 92a0f6d1577514d573f9722c366384d6
I20260619 16:44:32.631904 10242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:44:32.626515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:49529 (local address 127.0.97.62:36913)
0619 16:44:32.626708 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0619 16:44:32.626712 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:44:32.627979 (+ 1267us) server_negotiation.cc:407] Connection header received
0619 16:44:32.629621 (+ 1642us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:44:32.629626 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:44:32.629694 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:44:32.629813 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:44:32.631215 (+ 1402us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:44:32.631221 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:44:32.631226 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:44:32.631244 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:44:32.631273 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:44:32.631283 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:44:32.631386 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0619 16:44:32.631572 (+ 186us) server_negotiation.cc:299] Negotiation successful
0619 16:44:32.631633 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260619 16:44:32.632802 10652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:44:32.627001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36913 (local address 127.0.97.3:49529)
0619 16:44:32.627835 (+ 834us) negotiation.cc:107] Waiting for socket to connect
0619 16:44:32.627871 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0619 16:44:32.629363 (+ 1492us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:44:32.630038 (+ 675us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:44:32.630046 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:44:32.630164 (+ 118us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:44:32.630900 (+ 736us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 16:44:32.630903 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:44:32.630954 (+ 51us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 16:44:32.631065 (+ 111us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:44:32.631362 (+ 297us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:44:32.631372 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:44:32.631380 (+ 8us) client_negotiation.cc:769] Sending connection context
0619 16:44:32.631625 (+ 245us) client_negotiation.cc:240] Negotiation successful
0619 16:44:32.631884 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":394,"thread_start_us":232,"threads_started":1}
I20260619 16:44:32.633976 10650 heartbeater.cc:347] Connected to a master server at 127.0.97.62:36913
I20260619 16:44:32.634263 10650 heartbeater.cc:464] Registering TS with master...
I20260619 16:44:32.634807 10650 heartbeater.cc:511] Master 127.0.97.62:36913 requested a full tablet report, sending...
I20260619 16:44:32.636030 10203 ts_manager.cc:194] Registered new tserver with Master: 9d5c3422e2b44673a5363999a8fabdc8 (127.0.97.3:45377)
I20260619 16:44:32.644505 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260619 16:44:32.650221 10242 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:44:32.648758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49056 (local address 127.0.97.62:36913)
0619 16:44:32.648907 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0619 16:44:32.648912 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:44:32.649308 (+ 396us) server_negotiation.cc:407] Connection header received
0619 16:44:32.649456 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:44:32.649461 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:44:32.649516 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:44:32.649591 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:44:32.649964 (+ 373us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49056: BlockingRecv error: recv got EOF from 127.0.0.1:49056 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
I20260619 16:44:32.650736 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu pid 10263
I20260619 16:44:32.654126 10659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:44:32.651672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59570 (local address 127.0.97.1:38129)
0619 16:44:32.652131 (+ 459us) server_negotiation.cc:206] Beginning negotiation
0619 16:44:32.652138 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:44:32.652161 (+ 23us) server_negotiation.cc:407] Connection header received
0619 16:44:32.652212 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:44:32.652217 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:44:32.652412 (+ 195us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:44:32.652556 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:44:32.653448 (+ 892us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:44:32.653454 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:44:32.653464 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:44:32.653480 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:44:32.653517 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:44:32.653530 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:44:32.653649 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0619 16:44:32.653791 (+ 142us) server_negotiation.cc:299] Negotiation successful
0619 16:44:32.653882 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":335,"thread_start_us":213,"threads_started":1}
I20260619 16:44:32.655244 10385 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:44:32.733417 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu with pid 10263
I20260619 16:44:32.746611 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu pid 10393
I20260619 16:44:32.749831 10661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:44:32.747483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54000 (local address 127.0.97.2:44751)
0619 16:44:32.747977 (+ 494us) server_negotiation.cc:206] Beginning negotiation
0619 16:44:32.747989 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 16:44:32.748014 (+ 25us) server_negotiation.cc:407] Connection header received
0619 16:44:32.748074 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:44:32.748080 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:44:32.748290 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:44:32.748411 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:44:32.749179 (+ 768us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:44:32.749184 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:44:32.749196 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:44:32.749208 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:44:32.749254 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:44:32.749267 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:44:32.749364 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0619 16:44:32.749439 (+ 75us) server_negotiation.cc:299] Negotiation successful
0619 16:44:32.749521 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":364,"thread_start_us":240,"threads_started":1}
I20260619 16:44:32.750963 10515 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:44:32.829195 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu with pid 10393
I20260619 16:44:32.841367 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu pid 10523
I20260619 16:44:32.844586 10663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:44:32.842210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44222 (local address 127.0.97.3:45377)
0619 16:44:32.842741 (+ 531us) server_negotiation.cc:206] Beginning negotiation
0619 16:44:32.842748 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:44:32.842777 (+ 29us) server_negotiation.cc:407] Connection header received
0619 16:44:32.842836 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:44:32.842843 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:44:32.843046 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:44:32.843179 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:44:32.843996 (+ 817us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:44:32.844002 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:44:32.844014 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:44:32.844025 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:44:32.844064 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:44:32.844079 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:44:32.844226 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0619 16:44:32.844273 (+ 47us) server_negotiation.cc:299] Negotiation successful
0619 16:44:32.844350 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":387,"thread_start_us":244,"threads_started":1}
I20260619 16:44:32.845695 10645 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:44:32.925688 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu with pid 10523
I20260619 16:44:32.939718 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu pid 10173
I20260619 16:44:32.940835 10234 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:44:33.038134 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu with pid 10173
2026-06-19T16:44:33Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (2147 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887385903558-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 16:44:33 dist-test-slave-wvz7 krb5kdc[10668](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 16:44:33 dist-test-slave-wvz7 krb5kdc[10668](info): set up 2 sockets
Jun 19 16:44:33 dist-test-slave-wvz7 krb5kdc[10668](info): commencing operation
krb5kdc: starting...
W20260619 16:44:34.030377 388 mini_kdc.cc:121] Time spent starting KDC: real 0.957s user 0.000s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887385903558-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887385903558-388-0/krb5kdc/test-user.keytab.
Jun 19 16:44:34 dist-test-slave-wvz7 krb5kdc[10668](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887474, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T16:44:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T16:44:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887385903558-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887385903558-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260619 16:44:34.218689 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu
/tmp/dist-test-taskEs2EH8/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887385903558-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887385903558-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887385903558-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887385903558-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:36763
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887385903558-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39579
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:36763
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887385903558-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887385903558-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887385903558-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:44:34.525429 10681 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:44:34.525750 10681 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:44:34.525875 10681 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:44:34.533414 10681 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:44:34.533545 10681 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:44:34.533648 10681 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:44:34.533747 10681 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:44:34.533809 10681 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:44:34.543498 10681 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39579
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887385903558-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887385903558-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:36763
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887385903558-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:36763
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887385903558-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887385903558-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:44:34.544760 10681 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:44:34.546067 10681 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:44:34.556789 10687 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:44:34.556831 10686 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:44:34.556859 10689 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:44:34.557602 10681 server_base.cc:1034] running on GCE node
I20260619 16:44:34.558298 10681 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:44:34.559543 10681 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:44:34.560721 10681 hybrid_clock.cc:648] HybridClock initialized: now 1781887474560668 us; error 44 us; skew 500 ppm
I20260619 16:44:34.561030 10681 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:44:34 dist-test-slave-wvz7 krb5kdc[10668](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887474, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:44:34.563938 10681 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260619 16:44:34.565214 10681 webserver.cc:460] Webserver started at http://127.0.97.62:41869/ using document root <none> and password file <none>
I20260619 16:44:34.565874 10681 fs_manager.cc:359] Metadata directory not provided
I20260619 16:44:34.565989 10681 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:44:34.566256 10681 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:44:34.568912 10681 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887385903558-388-0/minicluster-data/master-0/data/instance:
uuid: "e2a0fedb35064355a5c2de8473dae593"
format_stamp: "Formatted at 2026-06-19 16:44:34 on dist-test-slave-wvz7"
server_key: "3fd702efee933cd110e5d5315484f7bd"
server_key_iv: "93cc81e99651620447801a173dbc193e"
server_key_version: "encryptionkey@0"
I20260619 16:44:34.569586 10681 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887385903558-388-0/minicluster-data/master-0/wal/instance:
uuid: "e2a0fedb35064355a5c2de8473dae593"
format_stamp: "Formatted at 2026-06-19 16:44:34 on dist-test-slave-wvz7"
server_key: "3fd702efee933cd110e5d5315484f7bd"
server_key_iv: "93cc81e99651620447801a173dbc193e"
server_key_version: "encryptionkey@0"
I20260619 16:44:34.574277 10681 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260619 16:44:34.577260 10696 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:44:34.578482 10681 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 16:44:34.578619 10681 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887385903558-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887385903558-388-0/minicluster-data/master-0/wal
uuid: "e2a0fedb35064355a5c2de8473dae593"
format_stamp: "Formatted at 2026-06-19 16:44:34 on dist-test-slave-wvz7"
server_key: "3fd702efee933cd110e5d5315484f7bd"
server_key_iv: "93cc81e99651620447801a173dbc193e"
server_key_version: "encryptionkey@0"
I20260619 16:44:34.578768 10681 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887385903558-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887385903558-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887385903558-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:44:34.592245 10681 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:44:34.596031 10681 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:44:34.596256 10681 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:44:34.611876 10681 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:36763
I20260619 16:44:34.611897 10747 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:36763 every 8 connection(s)
I20260619 16:44:34.613072 10681 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887385903558-388-0/minicluster-data/master-0/data/info.pb
I20260619 16:44:34.614111 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu as pid 10681
I20260619 16:44:34.614326 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887385903558-388-0/minicluster-data/master-0/wal/instance
I20260619 16:44:34.614935 388 external_mini_cluster.cc:1428] Setting key 15fd28c5c4b916fb3acfff1b7eaedd97
I20260619 16:44:34.616991 10748 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Jun 19 16:44:34 dist-test-slave-wvz7 krb5kdc[10668](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887474, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 16:44:34.628902 10748 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e2a0fedb35064355a5c2de8473dae593: Bootstrap starting.
I20260619 16:44:34.631891 10748 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e2a0fedb35064355a5c2de8473dae593: Neither blocks nor log segments found. Creating new log.
I20260619 16:44:34.632939 10748 log.cc:824] T 00000000000000000000000000000000 P e2a0fedb35064355a5c2de8473dae593: Log is configured to *not* fsync() on all Append() calls
I20260619 16:44:34.635553 10748 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e2a0fedb35064355a5c2de8473dae593: No bootstrap required, opened a new log
I20260619 16:44:34.636333 10751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:44:34.616732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54012 (local address 127.0.97.62:36763)
0619 16:44:34.617485 (+ 753us) server_negotiation.cc:206] Beginning negotiation
0619 16:44:34.617499 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 16:44:34.617544 (+ 45us) server_negotiation.cc:407] Connection header received
0619 16:44:34.619623 (+ 2079us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:44:34.619647 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:44:34.620124 (+ 477us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:44:34.620517 (+ 393us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:44:34.621672 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:44:34.623624 (+ 1952us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:44:34.624867 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:44:34.625830 (+ 963us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:44:34.626006 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:44:34.629672 (+ 3666us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:44:34.629678 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:44:34.629692 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:44:34.629726 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:44:34.632467 (+ 2741us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:44:34.633229 (+ 762us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:44:34.633235 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:44:34.633241 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:44:34.633406 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:44:34.633836 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:44:34.633842 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:44:34.633845 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:44:34.634357 (+ 512us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:44:34.634601 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0619 16:44:34.635029 (+ 428us) server_negotiation.cc:299] Negotiation successful
0619 16:44:34.635395 (+ 366us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":524,"thread_start_us":294,"threads_started":1}
I20260619 16:44:34.642781 10748 raft_consensus.cc:348] T 00000000000000000000000000000000 P e2a0fedb35064355a5c2de8473dae593 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2a0fedb35064355a5c2de8473dae593" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36763 } }
I20260619 16:44:34.643077 10748 raft_consensus.cc:374] T 00000000000000000000000000000000 P e2a0fedb35064355a5c2de8473dae593 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:44:34.643164 10748 raft_consensus.cc:729] T 00000000000000000000000000000000 P e2a0fedb35064355a5c2de8473dae593 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2a0fedb35064355a5c2de8473dae593, State: Initialized, Role: FOLLOWER
I20260619 16:44:34.643669 10748 consensus_queue.cc:260] T 00000000000000000000000000000000 P e2a0fedb35064355a5c2de8473dae593 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2a0fedb35064355a5c2de8473dae593" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36763 } }
I20260619 16:44:34.643930 10748 raft_consensus.cc:388] T 00000000000000000000000000000000 P e2a0fedb35064355a5c2de8473dae593 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:44:34.644050 10748 raft_consensus.cc:482] T 00000000000000000000000000000000 P e2a0fedb35064355a5c2de8473dae593 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:44:34.644265 10748 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e2a0fedb35064355a5c2de8473dae593 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:44:34.646250 10748 raft_consensus.cc:504] T 00000000000000000000000000000000 P e2a0fedb35064355a5c2de8473dae593 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2a0fedb35064355a5c2de8473dae593" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36763 } }
I20260619 16:44:34.646718 10748 leader_election.cc:302] T 00000000000000000000000000000000 P e2a0fedb35064355a5c2de8473dae593 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e2a0fedb35064355a5c2de8473dae593; no voters:
I20260619 16:44:34.647254 10748 leader_election.cc:288] T 00000000000000000000000000000000 P e2a0fedb35064355a5c2de8473dae593 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 16:44:34.647342 10753 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e2a0fedb35064355a5c2de8473dae593 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:44:34.648012 10753 raft_consensus.cc:686] T 00000000000000000000000000000000 P e2a0fedb35064355a5c2de8473dae593 [term 1 LEADER]: Becoming Leader. State: Replica: e2a0fedb35064355a5c2de8473dae593, State: Running, Role: LEADER
I20260619 16:44:34.648574 10753 consensus_queue.cc:237] T 00000000000000000000000000000000 P e2a0fedb35064355a5c2de8473dae593 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2a0fedb35064355a5c2de8473dae593" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36763 } }
I20260619 16:44:34.651963 10753 sys_catalog.cc:455] T 00000000000000000000000000000000 P e2a0fedb35064355a5c2de8473dae593 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e2a0fedb35064355a5c2de8473dae593. Latest consensus state: current_term: 1 leader_uuid: "e2a0fedb35064355a5c2de8473dae593" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2a0fedb35064355a5c2de8473dae593" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36763 } } }
I20260619 16:44:34.652487 10753 sys_catalog.cc:458] T 00000000000000000000000000000000 P e2a0fedb35064355a5c2de8473dae593 [sys.catalog]: This master's current role is: LEADER
I20260619 16:44:34.652292 10755 sys_catalog.cc:455] T 00000000000000000000000000000000 P e2a0fedb35064355a5c2de8473dae593 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e2a0fedb35064355a5c2de8473dae593" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2a0fedb35064355a5c2de8473dae593" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36763 } } }
I20260619 16:44:34.652843 10755 sys_catalog.cc:458] T 00000000000000000000000000000000 P e2a0fedb35064355a5c2de8473dae593 [sys.catalog]: This master's current role is: LEADER
I20260619 16:44:34.653630 10748 sys_catalog.cc:564] T 00000000000000000000000000000000 P e2a0fedb35064355a5c2de8473dae593 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:44:34.655095 10757 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:44:34.660229 10757 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:44:34.672583 10757 catalog_manager.cc:1347] Generated new cluster ID: 391db89c19374f658504cad9a113da32
I20260619 16:44:34.672705 10757 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:44:34.695508 10757 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 16:44:34.697329 10757 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:44:34.703958 10757 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e2a0fedb35064355a5c2de8473dae593: Generated new TSK 0
I20260619 16:44:34.704962 10757 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887385903558-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887385903558-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260619 16:44:34.761029 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu
/tmp/dist-test-taskEs2EH8/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887385903558-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887385903558-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887385903558-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887385903558-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36763
--keytab_file=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887385903558-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39579
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887385903558-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887385903558-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887385903558-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:44:35.061473 10774 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:44:35.061758 10774 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:44:35.061866 10774 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:44:35.068733 10774 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:44:35.068825 10774 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:44:35.068930 10774 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260619 16:44:35.077960 10774 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39579
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887385903558-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887385903558-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887385903558-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887385903558-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36763
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887385903558-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:44:35.079046 10774 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:44:35.080276 10774 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:44:35.092429 10779 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:44:35.092344 10782 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:44:35.092382 10780 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:44:35.093295 10774 server_base.cc:1034] running on GCE node
I20260619 16:44:35.093870 10774 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:44:35.094710 10774 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:44:35.095898 10774 hybrid_clock.cc:648] HybridClock initialized: now 1781887475095846 us; error 45 us; skew 500 ppm
I20260619 16:44:35.096180 10774 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:44:35 dist-test-slave-wvz7 krb5kdc[10668](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887475, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:44:35.099090 10774 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260619 16:44:35.100384 10774 webserver.cc:460] Webserver started at http://127.0.97.1:36697/ using document root <none> and password file <none>
I20260619 16:44:35.100976 10774 fs_manager.cc:359] Metadata directory not provided
I20260619 16:44:35.101073 10774 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:44:35.101291 10774 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:44:35.103940 10774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887385903558-388-0/minicluster-data/ts-0/data/instance:
uuid: "fdcb93d084a2480b8bf4f02b5b761d7e"
format_stamp: "Formatted at 2026-06-19 16:44:35 on dist-test-slave-wvz7"
server_key: "3f930f8c3f2d4095585857c2153c4629"
server_key_iv: "ad930d11a281b37c0258ae1f098f8bb0"
server_key_version: "encryptionkey@0"
I20260619 16:44:35.104584 10774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887385903558-388-0/minicluster-data/ts-0/wal/instance:
uuid: "fdcb93d084a2480b8bf4f02b5b761d7e"
format_stamp: "Formatted at 2026-06-19 16:44:35 on dist-test-slave-wvz7"
server_key: "3f930f8c3f2d4095585857c2153c4629"
server_key_iv: "ad930d11a281b37c0258ae1f098f8bb0"
server_key_version: "encryptionkey@0"
I20260619 16:44:35.109151 10774 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.005s
I20260619 16:44:35.112107 10789 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:44:35.113287 10774 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 16:44:35.113416 10774 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887385903558-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887385903558-388-0/minicluster-data/ts-0/wal
uuid: "fdcb93d084a2480b8bf4f02b5b761d7e"
format_stamp: "Formatted at 2026-06-19 16:44:35 on dist-test-slave-wvz7"
server_key: "3f930f8c3f2d4095585857c2153c4629"
server_key_iv: "ad930d11a281b37c0258ae1f098f8bb0"
server_key_version: "encryptionkey@0"
I20260619 16:44:35.113571 10774 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887385903558-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887385903558-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887385903558-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:44:35.132063 10774 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:44:35.135704 10774 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:44:35.135931 10774 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:44:35.136744 10774 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:44:35.138029 10774 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:44:35.138094 10774 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:44:35.138200 10774 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:44:35.138247 10774 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:44:35.167290 10774 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:37931
I20260619 16:44:35.167361 10901 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:37931 every 8 connection(s)
I20260619 16:44:35.168468 10774 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887385903558-388-0/minicluster-data/ts-0/data/info.pb
I20260619 16:44:35.169934 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu as pid 10774
I20260619 16:44:35.170179 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887385903558-388-0/minicluster-data/ts-0/wal/instance
I20260619 16:44:35.170837 388 external_mini_cluster.cc:1428] Setting key 15b925a615076abf72727de83f166c03
Jun 19 16:44:35 dist-test-slave-wvz7 krb5kdc[10668](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887475, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 16:44:35.187574 10904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:44:35.171095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:45999 (local address 127.0.97.62:36763)
0619 16:44:35.171528 (+ 433us) server_negotiation.cc:206] Beginning negotiation
0619 16:44:35.171535 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:44:35.172785 (+ 1250us) server_negotiation.cc:407] Connection header received
0619 16:44:35.174353 (+ 1568us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:44:35.174358 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:44:35.174434 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:44:35.174533 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:44:35.175981 (+ 1448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:44:35.176674 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:44:35.178245 (+ 1571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:44:35.178627 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:44:35.178733 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:44:35.182363 (+ 3630us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:44:35.182368 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:44:35.182371 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:44:35.182402 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:44:35.184758 (+ 2356us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:44:35.185406 (+ 648us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:44:35.185410 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:44:35.185413 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:44:35.185520 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:44:35.185960 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:44:35.185964 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:44:35.185966 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:44:35.186234 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:44:35.186358 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0619 16:44:35.186999 (+ 641us) server_negotiation.cc:299] Negotiation successful
0619 16:44:35.187225 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":313,"thread_start_us":207,"threads_started":1}
I20260619 16:44:35.188467 10905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:44:35.171599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36763 (local address 127.0.97.1:45999)
0619 16:44:35.172341 (+ 742us) negotiation.cc:107] Waiting for socket to connect
0619 16:44:35.172399 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0619 16:44:35.174104 (+ 1705us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:44:35.174771 (+ 667us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:44:35.174779 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:44:35.175279 (+ 500us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:44:35.175828 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:44:35.175845 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:44:35.176808 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:44:35.176812 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:44:35.178084 (+ 1272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:44:35.178096 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:44:35.178829 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:44:35.178836 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:44:35.179065 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:44:35.179693 (+ 628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:44:35.179723 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:44:35.182150 (+ 2427us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:44:35.184882 (+ 2732us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:44:35.184890 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:44:35.184906 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:44:35.185294 (+ 388us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:44:35.185622 (+ 328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:44:35.185627 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:44:35.185631 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:44:35.185849 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:44:35.186345 (+ 496us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:44:35.186352 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:44:35.186672 (+ 320us) client_negotiation.cc:769] Sending connection context
0619 16:44:35.186894 (+ 222us) client_negotiation.cc:240] Negotiation successful
0619 16:44:35.187392 (+ 498us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":477,"thread_start_us":269,"threads_started":1}
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260619 16:44:35.189956 10902 heartbeater.cc:347] Connected to a master server at 127.0.97.62:36763
I20260619 16:44:35.190271 10902 heartbeater.cc:464] Registering TS with master...
I20260619 16:44:35.190922 10902 heartbeater.cc:511] Master 127.0.97.62:36763 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260619 16:44:35.193015 10713 ts_manager.cc:194] Registered new tserver with Master: fdcb93d084a2480b8bf4f02b5b761d7e (127.0.97.1:37931)
I20260619 16:44:35.194881 10713 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:45999
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887385903558-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887385903558-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260619 16:44:35.214447 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu
/tmp/dist-test-taskEs2EH8/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887385903558-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887385903558-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887385903558-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887385903558-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36763
--keytab_file=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887385903558-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39579
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887385903558-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887385903558-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887385903558-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:44:35.508715 10908 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:44:35.508967 10908 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:44:35.509056 10908 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:44:35.517026 10908 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:44:35.517127 10908 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:44:35.517242 10908 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260619 16:44:35.526520 10908 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39579
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887385903558-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887385903558-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887385903558-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887385903558-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36763
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887385903558-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:44:35.527631 10908 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:44:35.528832 10908 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:44:35.541993 10913 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:44:35.541958 10916 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:44:35.542029 10914 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:44:35.542609 10908 server_base.cc:1034] running on GCE node
I20260619 16:44:35.543114 10908 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:44:35.543901 10908 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:44:35.545075 10908 hybrid_clock.cc:648] HybridClock initialized: now 1781887475545022 us; error 48 us; skew 500 ppm
I20260619 16:44:35.545351 10908 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:44:35 dist-test-slave-wvz7 krb5kdc[10668](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887475, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:44:35.548488 10908 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260619 16:44:35.550010 10908 webserver.cc:460] Webserver started at http://127.0.97.2:44583/ using document root <none> and password file <none>
I20260619 16:44:35.550700 10908 fs_manager.cc:359] Metadata directory not provided
I20260619 16:44:35.550820 10908 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:44:35.551106 10908 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:44:35.554178 10908 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887385903558-388-0/minicluster-data/ts-1/data/instance:
uuid: "bb6166b6a1fd4de0876f490bb2ccb2a7"
format_stamp: "Formatted at 2026-06-19 16:44:35 on dist-test-slave-wvz7"
server_key: "9f0552a2d390354b5a740f2ca9b6670d"
server_key_iv: "7f01ff27e0ce7f073d5a263a67b54636"
server_key_version: "encryptionkey@0"
I20260619 16:44:35.554992 10908 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887385903558-388-0/minicluster-data/ts-1/wal/instance:
uuid: "bb6166b6a1fd4de0876f490bb2ccb2a7"
format_stamp: "Formatted at 2026-06-19 16:44:35 on dist-test-slave-wvz7"
server_key: "9f0552a2d390354b5a740f2ca9b6670d"
server_key_iv: "7f01ff27e0ce7f073d5a263a67b54636"
server_key_version: "encryptionkey@0"
I20260619 16:44:35.560143 10908 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260619 16:44:35.563441 10923 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:44:35.564695 10908 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 16:44:35.564832 10908 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887385903558-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887385903558-388-0/minicluster-data/ts-1/wal
uuid: "bb6166b6a1fd4de0876f490bb2ccb2a7"
format_stamp: "Formatted at 2026-06-19 16:44:35 on dist-test-slave-wvz7"
server_key: "9f0552a2d390354b5a740f2ca9b6670d"
server_key_iv: "7f01ff27e0ce7f073d5a263a67b54636"
server_key_version: "encryptionkey@0"
I20260619 16:44:35.565006 10908 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887385903558-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887385903558-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887385903558-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:44:35.593001 10908 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:44:35.597157 10908 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:44:35.597430 10908 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:44:35.598413 10908 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:44:35.599772 10908 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:44:35.599862 10908 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:44:35.599982 10908 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:44:35.600033 10908 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:44:35.628927 10908 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:46353
I20260619 16:44:35.628943 11035 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:46353 every 8 connection(s)
I20260619 16:44:35.630385 10908 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887385903558-388-0/minicluster-data/ts-1/data/info.pb
I20260619 16:44:35.640580 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu as pid 10908
I20260619 16:44:35.640812 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887385903558-388-0/minicluster-data/ts-1/wal/instance
I20260619 16:44:35.641214 388 external_mini_cluster.cc:1428] Setting key b52f7888f9ba1f61705e2506839c4d27
Jun 19 16:44:35 dist-test-slave-wvz7 krb5kdc[10668](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887475, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 16:44:35.648516 10904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:44:35.632779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:55523 (local address 127.0.97.62:36763)
0619 16:44:35.632974 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0619 16:44:35.632980 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:44:35.634047 (+ 1067us) server_negotiation.cc:407] Connection header received
0619 16:44:35.635755 (+ 1708us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:44:35.635760 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:44:35.635837 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:44:35.635923 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:44:35.637284 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:44:35.638058 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:44:35.639482 (+ 1424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:44:35.639891 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:44:35.639993 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:44:35.643484 (+ 3491us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:44:35.643488 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:44:35.643492 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:44:35.643521 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:44:35.645857 (+ 2336us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:44:35.646572 (+ 715us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:44:35.646575 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:44:35.646578 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:44:35.646716 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:44:35.647090 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:44:35.647093 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:44:35.647095 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:44:35.647326 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:44:35.647427 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0619 16:44:35.648134 (+ 707us) server_negotiation.cc:299] Negotiation successful
0619 16:44:35.648293 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260619 16:44:35.649699 11038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:44:35.633217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36763 (local address 127.0.97.2:55523)
0619 16:44:35.633902 (+ 685us) negotiation.cc:107] Waiting for socket to connect
0619 16:44:35.633946 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0619 16:44:35.635515 (+ 1569us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:44:35.636147 (+ 632us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:44:35.636155 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:44:35.636596 (+ 441us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:44:35.637139 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:44:35.637155 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:44:35.638193 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:44:35.638197 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:44:35.639356 (+ 1159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:44:35.639368 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:44:35.640081 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:44:35.640088 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:44:35.640302 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:44:35.640919 (+ 617us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:44:35.640947 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:44:35.643301 (+ 2354us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:44:35.645974 (+ 2673us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:44:35.645982 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:44:35.645995 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:44:35.646483 (+ 488us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:44:35.646803 (+ 320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:44:35.646807 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:44:35.646810 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:44:35.647009 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:44:35.647424 (+ 415us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:44:35.647431 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:44:35.647807 (+ 376us) client_negotiation.cc:769] Sending connection context
0619 16:44:35.648067 (+ 260us) client_negotiation.cc:240] Negotiation successful
0619 16:44:35.648520 (+ 453us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":444,"thread_start_us":252,"threads_started":1}
I20260619 16:44:35.651149 11036 heartbeater.cc:347] Connected to a master server at 127.0.97.62:36763
I20260619 16:44:35.651397 11036 heartbeater.cc:464] Registering TS with master...
I20260619 16:44:35.652016 11036 heartbeater.cc:511] Master 127.0.97.62:36763 requested a full tablet report, sending...
I20260619 16:44:35.653226 10713 ts_manager.cc:194] Registered new tserver with Master: bb6166b6a1fd4de0876f490bb2ccb2a7 (127.0.97.2:46353)
I20260619 16:44:35.654027 10713 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:55523
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887385903558-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887385903558-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260619 16:44:35.686496 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu
/tmp/dist-test-taskEs2EH8/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887385903558-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887385903558-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887385903558-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887385903558-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36763
--keytab_file=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887385903558-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39579
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887385903558-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887385903558-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887385903558-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:44:35.989372 11041 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:44:35.989648 11041 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:44:35.989758 11041 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:44:35.996702 11041 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:44:35.996815 11041 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:44:35.996929 11041 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260619 16:44:36.006578 11041 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39579
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887385903558-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887385903558-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887385903558-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887385903558-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36763
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887385903558-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:44:36.007782 11041 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:44:36.009007 11041 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:44:36.021718 11046 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:44:36.021703 11047 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:44:36.021770 11049 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:44:36.022538 11041 server_base.cc:1034] running on GCE node
I20260619 16:44:36.023211 11041 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:44:36.024118 11041 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:44:36.025324 11041 hybrid_clock.cc:648] HybridClock initialized: now 1781887476025275 us; error 45 us; skew 500 ppm
I20260619 16:44:36.025637 11041 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:44:36 dist-test-slave-wvz7 krb5kdc[10668](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887476, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:44:36.028681 11041 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260619 16:44:36.030156 11041 webserver.cc:460] Webserver started at http://127.0.97.3:34949/ using document root <none> and password file <none>
I20260619 16:44:36.030848 11041 fs_manager.cc:359] Metadata directory not provided
I20260619 16:44:36.030958 11041 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:44:36.031236 11041 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:44:36.034082 11041 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887385903558-388-0/minicluster-data/ts-2/data/instance:
uuid: "2d262a9be62a46429b37d460bb96bd44"
format_stamp: "Formatted at 2026-06-19 16:44:36 on dist-test-slave-wvz7"
server_key: "173db8fc5cbba581f81068ffff7435c7"
server_key_iv: "d6d141428944020e9b27273d7868f635"
server_key_version: "encryptionkey@0"
I20260619 16:44:36.034842 11041 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887385903558-388-0/minicluster-data/ts-2/wal/instance:
uuid: "2d262a9be62a46429b37d460bb96bd44"
format_stamp: "Formatted at 2026-06-19 16:44:36 on dist-test-slave-wvz7"
server_key: "173db8fc5cbba581f81068ffff7435c7"
server_key_iv: "d6d141428944020e9b27273d7868f635"
server_key_version: "encryptionkey@0"
I20260619 16:44:36.039805 11041 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260619 16:44:36.043063 11056 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:44:36.044510 11041 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 16:44:36.044664 11041 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887385903558-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887385903558-388-0/minicluster-data/ts-2/wal
uuid: "2d262a9be62a46429b37d460bb96bd44"
format_stamp: "Formatted at 2026-06-19 16:44:36 on dist-test-slave-wvz7"
server_key: "173db8fc5cbba581f81068ffff7435c7"
server_key_iv: "d6d141428944020e9b27273d7868f635"
server_key_version: "encryptionkey@0"
I20260619 16:44:36.044850 11041 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887385903558-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887385903558-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887385903558-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:44:36.059482 11041 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:44:36.063019 11041 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:44:36.063270 11041 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:44:36.064143 11041 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:44:36.065403 11041 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:44:36.065485 11041 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:44:36.065574 11041 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:44:36.065620 11041 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:44:36.094357 11041 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:37847
I20260619 16:44:36.094400 11168 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:37847 every 8 connection(s)
I20260619 16:44:36.095505 11041 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887385903558-388-0/minicluster-data/ts-2/data/info.pb
I20260619 16:44:36.105319 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu as pid 11041
I20260619 16:44:36.105516 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887385903558-388-0/minicluster-data/ts-2/wal/instance
I20260619 16:44:36.106060 388 external_mini_cluster.cc:1428] Setting key 3d1792d676918fabd23a42d5d55e1fed
Jun 19 16:44:36 dist-test-slave-wvz7 krb5kdc[10668](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887476, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 16:44:36.113490 10904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:44:36.097770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:42185 (local address 127.0.97.62:36763)
0619 16:44:36.097964 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0619 16:44:36.097970 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:44:36.099113 (+ 1143us) server_negotiation.cc:407] Connection header received
0619 16:44:36.100860 (+ 1747us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:44:36.100864 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:44:36.100926 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:44:36.101009 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:44:36.102425 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:44:36.103131 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:44:36.104607 (+ 1476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:44:36.104999 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:44:36.105089 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:44:36.108641 (+ 3552us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:44:36.108647 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:44:36.108652 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:44:36.108700 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:44:36.111141 (+ 2441us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:44:36.111765 (+ 624us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:44:36.111768 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:44:36.111771 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:44:36.111872 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:44:36.112197 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:44:36.112199 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:44:36.112201 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:44:36.112442 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:44:36.112541 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0619 16:44:36.113101 (+ 560us) server_negotiation.cc:299] Negotiation successful
0619 16:44:36.113250 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260619 16:44:36.114427 11171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:44:36.098294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36763 (local address 127.0.97.3:42185)
0619 16:44:36.098950 (+ 656us) negotiation.cc:107] Waiting for socket to connect
0619 16:44:36.098991 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0619 16:44:36.100615 (+ 1624us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:44:36.101187 (+ 572us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:44:36.101194 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:44:36.101714 (+ 520us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:44:36.102277 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:44:36.102294 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:44:36.103276 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:44:36.103280 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:44:36.104457 (+ 1177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:44:36.104468 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:44:36.105207 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:44:36.105214 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:44:36.105530 (+ 316us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:44:36.106330 (+ 800us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:44:36.106351 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:44:36.108480 (+ 2129us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:44:36.111276 (+ 2796us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:44:36.111282 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:44:36.111293 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:44:36.111676 (+ 383us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:44:36.111944 (+ 268us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:44:36.111947 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:44:36.111949 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:44:36.112119 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:44:36.112524 (+ 405us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:44:36.112529 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:44:36.112807 (+ 278us) client_negotiation.cc:769] Sending connection context
0619 16:44:36.113027 (+ 220us) client_negotiation.cc:240] Negotiation successful
0619 16:44:36.113373 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":435,"thread_start_us":252,"threads_started":1}
I20260619 16:44:36.115643 11169 heartbeater.cc:347] Connected to a master server at 127.0.97.62:36763
I20260619 16:44:36.115907 11169 heartbeater.cc:464] Registering TS with master...
I20260619 16:44:36.116624 11169 heartbeater.cc:511] Master 127.0.97.62:36763 requested a full tablet report, sending...
I20260619 16:44:36.117937 10713 ts_manager.cc:194] Registered new tserver with Master: 2d262a9be62a46429b37d460bb96bd44 (127.0.97.3:37847)
I20260619 16:44:36.118672 10713 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:42185
I20260619 16:44:36.121085 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 16:44:36.135973 10904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:44:36.125288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54024 (local address 127.0.97.62:36763)
0619 16:44:36.125475 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0619 16:44:36.125482 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:44:36.125838 (+ 356us) server_negotiation.cc:407] Connection header received
0619 16:44:36.125995 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:44:36.125999 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:44:36.126071 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:44:36.126151 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:44:36.127364 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:44:36.128039 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:44:36.129204 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:44:36.129590 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:44:36.129683 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:44:36.131602 (+ 1919us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:44:36.131606 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:44:36.131609 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:44:36.131636 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:44:36.133919 (+ 2283us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:44:36.134473 (+ 554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:44:36.134476 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:44:36.134479 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:44:36.134575 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:44:36.134824 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:44:36.134827 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:44:36.134829 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:44:36.135085 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:44:36.135266 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0619 16:44:36.135560 (+ 294us) server_negotiation.cc:299] Negotiation successful
0619 16:44:36.135700 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260619 16:44:36.139489 10713 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54024:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 16:44:36.145946 10713 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 16:44:36.167455 11185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:44:36.157247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:37847 (local address 127.0.0.1:47676)
0619 16:44:36.160257 (+ 3010us) negotiation.cc:107] Waiting for socket to connect
0619 16:44:36.160277 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 16:44:36.160390 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:44:36.160960 (+ 570us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:44:36.160965 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:44:36.160999 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:44:36.161392 (+ 393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:44:36.161405 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:44:36.163092 (+ 1687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:44:36.163097 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:44:36.164934 (+ 1837us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:44:36.164949 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:44:36.166706 (+ 1757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:44:36.166713 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:44:36.166862 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:44:36.166891 (+ 29us) client_negotiation.cc:769] Sending connection context
0619 16:44:36.166961 (+ 70us) client_negotiation.cc:240] Negotiation successful
0619 16:44:36.167084 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2884,"mutex_wait_us":6,"thread_start_us":260,"threads_started":1}
I20260619 16:44:36.167924 11184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:44:36.157985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47676 (local address 127.0.97.3:37847)
0619 16:44:36.159600 (+ 1615us) server_negotiation.cc:206] Beginning negotiation
0619 16:44:36.159608 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 16:44:36.160379 (+ 771us) server_negotiation.cc:407] Connection header received
0619 16:44:36.160523 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:44:36.160531 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:44:36.160771 (+ 240us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:44:36.161577 (+ 806us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:44:36.161801 (+ 224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:44:36.162965 (+ 1164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:44:36.165101 (+ 2136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:44:36.166470 (+ 1369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:44:36.167190 (+ 720us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:44:36.167390 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0619 16:44:36.167457 (+ 67us) server_negotiation.cc:299] Negotiation successful
0619 16:44:36.167608 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1470,"thread_start_us":1326,"threads_started":1}
I20260619 16:44:36.167076 11182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:44:36.157045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:37931 (local address 127.0.0.1:39118)
0619 16:44:36.157898 (+ 853us) negotiation.cc:107] Waiting for socket to connect
0619 16:44:36.157943 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0619 16:44:36.158118 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:44:36.159740 (+ 1622us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:44:36.159747 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:44:36.159777 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:44:36.160219 (+ 442us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:44:36.160238 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:44:36.162187 (+ 1949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:44:36.162193 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:44:36.164619 (+ 2426us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:44:36.164633 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:44:36.165994 (+ 1361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:44:36.166000 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:44:36.166186 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:44:36.166234 (+ 48us) client_negotiation.cc:769] Sending connection context
0619 16:44:36.166365 (+ 131us) client_negotiation.cc:240] Negotiation successful
0619 16:44:36.166545 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":586,"mutex_wait_us":47,"thread_start_us":3545,"threads_started":1}
I20260619 16:44:36.170710 11181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:44:36.158389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:46353 (local address 127.0.0.1:40548)
0619 16:44:36.160850 (+ 2461us) negotiation.cc:107] Waiting for socket to connect
0619 16:44:36.160865 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 16:44:36.161043 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:44:36.161865 (+ 822us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:44:36.161869 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:44:36.161892 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:44:36.162251 (+ 359us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:44:36.162262 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:44:36.164079 (+ 1817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:44:36.164085 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:44:36.166095 (+ 2010us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:44:36.166108 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:44:36.169916 (+ 3808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:44:36.169923 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:44:36.170108 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:44:36.170141 (+ 33us) client_negotiation.cc:769] Sending connection context
0619 16:44:36.170247 (+ 106us) client_negotiation.cc:240] Negotiation successful
0619 16:44:36.170367 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2336,"mutex_wait_us":3,"thread_start_us":1621,"threads_started":1}
I20260619 16:44:36.170758 11186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:44:36.158860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40548 (local address 127.0.97.2:46353)
0619 16:44:36.159865 (+ 1005us) server_negotiation.cc:206] Beginning negotiation
0619 16:44:36.159874 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 16:44:36.160948 (+ 1074us) server_negotiation.cc:407] Connection header received
0619 16:44:36.161220 (+ 272us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:44:36.161228 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:44:36.161442 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:44:36.161607 (+ 165us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:44:36.162426 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:44:36.163907 (+ 1481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:44:36.166222 (+ 2315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:44:36.167593 (+ 1371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:44:36.167696 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:44:36.167876 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0619 16:44:36.170286 (+ 2410us) server_negotiation.cc:299] Negotiation successful
0619 16:44:36.170455 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":869,"thread_start_us":740,"threads_started":1}
I20260619 16:44:36.171394 11183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:44:36.157602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39118 (local address 127.0.97.1:37931)
0619 16:44:36.158591 (+ 989us) server_negotiation.cc:206] Beginning negotiation
0619 16:44:36.158600 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 16:44:36.158631 (+ 31us) server_negotiation.cc:407] Connection header received
0619 16:44:36.158746 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:44:36.158754 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:44:36.159010 (+ 256us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:44:36.160429 (+ 1419us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:44:36.160639 (+ 210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:44:36.162034 (+ 1395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:44:36.164771 (+ 2737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:44:36.165811 (+ 1040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:44:36.165921 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:44:36.166125 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0619 16:44:36.166426 (+ 301us) server_negotiation.cc:299] Negotiation successful
0619 16:44:36.171059 (+ 4633us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":841,"thread_start_us":754,"threads_started":1}
I20260619 16:44:36.171733 11104 tablet_service.cc:1467] Processing CreateTablet for tablet d53d03e6e3d74e129ae000b1bf4d99c1 (DEFAULT_TABLE table=test-table [id=504e126c62464b889bbbf81d27ecaf3b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:44:36.173113 11104 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d53d03e6e3d74e129ae000b1bf4d99c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:44:36.175000 10837 tablet_service.cc:1467] Processing CreateTablet for tablet d53d03e6e3d74e129ae000b1bf4d99c1 (DEFAULT_TABLE table=test-table [id=504e126c62464b889bbbf81d27ecaf3b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:44:36.176296 10837 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d53d03e6e3d74e129ae000b1bf4d99c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:44:36.176409 10971 tablet_service.cc:1467] Processing CreateTablet for tablet d53d03e6e3d74e129ae000b1bf4d99c1 (DEFAULT_TABLE table=test-table [id=504e126c62464b889bbbf81d27ecaf3b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:44:36.177623 10971 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d53d03e6e3d74e129ae000b1bf4d99c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:44:36.186971 11187 tablet_bootstrap.cc:492] T d53d03e6e3d74e129ae000b1bf4d99c1 P 2d262a9be62a46429b37d460bb96bd44: Bootstrap starting.
I20260619 16:44:36.189776 11188 tablet_bootstrap.cc:492] T d53d03e6e3d74e129ae000b1bf4d99c1 P fdcb93d084a2480b8bf4f02b5b761d7e: Bootstrap starting.
I20260619 16:44:36.190671 11187 tablet_bootstrap.cc:654] T d53d03e6e3d74e129ae000b1bf4d99c1 P 2d262a9be62a46429b37d460bb96bd44: Neither blocks nor log segments found. Creating new log.
I20260619 16:44:36.190891 11189 tablet_bootstrap.cc:492] T d53d03e6e3d74e129ae000b1bf4d99c1 P bb6166b6a1fd4de0876f490bb2ccb2a7: Bootstrap starting.
I20260619 16:44:36.192637 11187 log.cc:824] T d53d03e6e3d74e129ae000b1bf4d99c1 P 2d262a9be62a46429b37d460bb96bd44: Log is configured to *not* fsync() on all Append() calls
I20260619 16:44:36.193043 11188 tablet_bootstrap.cc:654] T d53d03e6e3d74e129ae000b1bf4d99c1 P fdcb93d084a2480b8bf4f02b5b761d7e: Neither blocks nor log segments found. Creating new log.
I20260619 16:44:36.193811 11189 tablet_bootstrap.cc:654] T d53d03e6e3d74e129ae000b1bf4d99c1 P bb6166b6a1fd4de0876f490bb2ccb2a7: Neither blocks nor log segments found. Creating new log.
I20260619 16:44:36.194196 11188 log.cc:824] T d53d03e6e3d74e129ae000b1bf4d99c1 P fdcb93d084a2480b8bf4f02b5b761d7e: Log is configured to *not* fsync() on all Append() calls
I20260619 16:44:36.194797 11189 log.cc:824] T d53d03e6e3d74e129ae000b1bf4d99c1 P bb6166b6a1fd4de0876f490bb2ccb2a7: Log is configured to *not* fsync() on all Append() calls
I20260619 16:44:36.195776 11187 tablet_bootstrap.cc:492] T d53d03e6e3d74e129ae000b1bf4d99c1 P 2d262a9be62a46429b37d460bb96bd44: No bootstrap required, opened a new log
I20260619 16:44:36.196043 11187 ts_tablet_manager.cc:1397] T d53d03e6e3d74e129ae000b1bf4d99c1 P 2d262a9be62a46429b37d460bb96bd44: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.000s
I20260619 16:44:36.196867 11188 tablet_bootstrap.cc:492] T d53d03e6e3d74e129ae000b1bf4d99c1 P fdcb93d084a2480b8bf4f02b5b761d7e: No bootstrap required, opened a new log
I20260619 16:44:36.197125 11188 ts_tablet_manager.cc:1397] T d53d03e6e3d74e129ae000b1bf4d99c1 P fdcb93d084a2480b8bf4f02b5b761d7e: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260619 16:44:36.197508 11189 tablet_bootstrap.cc:492] T d53d03e6e3d74e129ae000b1bf4d99c1 P bb6166b6a1fd4de0876f490bb2ccb2a7: No bootstrap required, opened a new log
I20260619 16:44:36.197762 11189 ts_tablet_manager.cc:1397] T d53d03e6e3d74e129ae000b1bf4d99c1 P bb6166b6a1fd4de0876f490bb2ccb2a7: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.004s
I20260619 16:44:36.197824 10902 heartbeater.cc:503] Master 127.0.97.62:36763 was elected leader, sending a full tablet report...
I20260619 16:44:36.205231 11187 raft_consensus.cc:348] T d53d03e6e3d74e129ae000b1bf4d99c1 P 2d262a9be62a46429b37d460bb96bd44 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdcb93d084a2480b8bf4f02b5b761d7e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37931 } } peers { permanent_uuid: "2d262a9be62a46429b37d460bb96bd44" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37847 } } peers { permanent_uuid: "bb6166b6a1fd4de0876f490bb2ccb2a7" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46353 } }
I20260619 16:44:36.205639 11187 raft_consensus.cc:374] T d53d03e6e3d74e129ae000b1bf4d99c1 P 2d262a9be62a46429b37d460bb96bd44 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:44:36.205771 11187 raft_consensus.cc:729] T d53d03e6e3d74e129ae000b1bf4d99c1 P 2d262a9be62a46429b37d460bb96bd44 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d262a9be62a46429b37d460bb96bd44, State: Initialized, Role: FOLLOWER
I20260619 16:44:36.206344 11187 consensus_queue.cc:260] T d53d03e6e3d74e129ae000b1bf4d99c1 P 2d262a9be62a46429b37d460bb96bd44 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdcb93d084a2480b8bf4f02b5b761d7e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37931 } } peers { permanent_uuid: "2d262a9be62a46429b37d460bb96bd44" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37847 } } peers { permanent_uuid: "bb6166b6a1fd4de0876f490bb2ccb2a7" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46353 } }
I20260619 16:44:36.207296 11169 heartbeater.cc:503] Master 127.0.97.62:36763 was elected leader, sending a full tablet report...
I20260619 16:44:36.207710 11188 raft_consensus.cc:348] T d53d03e6e3d74e129ae000b1bf4d99c1 P fdcb93d084a2480b8bf4f02b5b761d7e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdcb93d084a2480b8bf4f02b5b761d7e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37931 } } peers { permanent_uuid: "2d262a9be62a46429b37d460bb96bd44" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37847 } } peers { permanent_uuid: "bb6166b6a1fd4de0876f490bb2ccb2a7" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46353 } }
I20260619 16:44:36.207855 11189 raft_consensus.cc:348] T d53d03e6e3d74e129ae000b1bf4d99c1 P bb6166b6a1fd4de0876f490bb2ccb2a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdcb93d084a2480b8bf4f02b5b761d7e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37931 } } peers { permanent_uuid: "2d262a9be62a46429b37d460bb96bd44" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37847 } } peers { permanent_uuid: "bb6166b6a1fd4de0876f490bb2ccb2a7" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46353 } }
I20260619 16:44:36.208120 11188 raft_consensus.cc:374] T d53d03e6e3d74e129ae000b1bf4d99c1 P fdcb93d084a2480b8bf4f02b5b761d7e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:44:36.208233 11189 raft_consensus.cc:374] T d53d03e6e3d74e129ae000b1bf4d99c1 P bb6166b6a1fd4de0876f490bb2ccb2a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:44:36.208228 11188 raft_consensus.cc:729] T d53d03e6e3d74e129ae000b1bf4d99c1 P fdcb93d084a2480b8bf4f02b5b761d7e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fdcb93d084a2480b8bf4f02b5b761d7e, State: Initialized, Role: FOLLOWER
I20260619 16:44:36.208333 11189 raft_consensus.cc:729] T d53d03e6e3d74e129ae000b1bf4d99c1 P bb6166b6a1fd4de0876f490bb2ccb2a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb6166b6a1fd4de0876f490bb2ccb2a7, State: Initialized, Role: FOLLOWER
I20260619 16:44:36.208788 11189 consensus_queue.cc:260] T d53d03e6e3d74e129ae000b1bf4d99c1 P bb6166b6a1fd4de0876f490bb2ccb2a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdcb93d084a2480b8bf4f02b5b761d7e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37931 } } peers { permanent_uuid: "2d262a9be62a46429b37d460bb96bd44" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37847 } } peers { permanent_uuid: "bb6166b6a1fd4de0876f490bb2ccb2a7" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46353 } }
I20260619 16:44:36.208758 11188 consensus_queue.cc:260] T d53d03e6e3d74e129ae000b1bf4d99c1 P fdcb93d084a2480b8bf4f02b5b761d7e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdcb93d084a2480b8bf4f02b5b761d7e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37931 } } peers { permanent_uuid: "2d262a9be62a46429b37d460bb96bd44" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37847 } } peers { permanent_uuid: "bb6166b6a1fd4de0876f490bb2ccb2a7" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46353 } }
I20260619 16:44:36.209007 11187 ts_tablet_manager.cc:1428] T d53d03e6e3d74e129ae000b1bf4d99c1 P 2d262a9be62a46429b37d460bb96bd44: Time spent starting tablet: real 0.013s user 0.012s sys 0.000s
I20260619 16:44:36.210335 11036 heartbeater.cc:503] Master 127.0.97.62:36763 was elected leader, sending a full tablet report...
I20260619 16:44:36.210825 11189 ts_tablet_manager.cc:1428] T d53d03e6e3d74e129ae000b1bf4d99c1 P bb6166b6a1fd4de0876f490bb2ccb2a7: Time spent starting tablet: real 0.013s user 0.008s sys 0.003s
I20260619 16:44:36.211534 11188 ts_tablet_manager.cc:1428] T d53d03e6e3d74e129ae000b1bf4d99c1 P fdcb93d084a2480b8bf4f02b5b761d7e: Time spent starting tablet: real 0.014s user 0.014s sys 0.000s
I20260619 16:44:36.244184 11193 raft_consensus.cc:482] T d53d03e6e3d74e129ae000b1bf4d99c1 P 2d262a9be62a46429b37d460bb96bd44 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 16:44:36.244524 11193 raft_consensus.cc:504] T d53d03e6e3d74e129ae000b1bf4d99c1 P 2d262a9be62a46429b37d460bb96bd44 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdcb93d084a2480b8bf4f02b5b761d7e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37931 } } peers { permanent_uuid: "2d262a9be62a46429b37d460bb96bd44" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37847 } } peers { permanent_uuid: "bb6166b6a1fd4de0876f490bb2ccb2a7" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46353 } }
I20260619 16:44:36.246331 11193 leader_election.cc:288] T d53d03e6e3d74e129ae000b1bf4d99c1 P 2d262a9be62a46429b37d460bb96bd44 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fdcb93d084a2480b8bf4f02b5b761d7e (127.0.97.1:37931), bb6166b6a1fd4de0876f490bb2ccb2a7 (127.0.97.2:46353)
I20260619 16:44:36.252856 11171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:44:36.246497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:37931 (local address 127.0.97.3:58843)
0619 16:44:36.246807 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0619 16:44:36.246827 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 16:44:36.246952 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:44:36.247491 (+ 539us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:44:36.247496 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:44:36.247524 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:44:36.247709 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:44:36.247718 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:44:36.248952 (+ 1234us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:44:36.248956 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:44:36.250904 (+ 1948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:44:36.250918 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:44:36.252281 (+ 1363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:44:36.252285 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:44:36.252410 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:44:36.252433 (+ 23us) client_negotiation.cc:769] Sending connection context
0619 16:44:36.252507 (+ 74us) client_negotiation.cc:240] Negotiation successful
0619 16:44:36.252599 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":180}
I20260619 16:44:36.252885 11183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:44:36.246731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:58843 (local address 127.0.97.1:37931)
0619 16:44:36.246912 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0619 16:44:36.246917 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:44:36.246934 (+ 17us) server_negotiation.cc:407] Connection header received
0619 16:44:36.247266 (+ 332us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:44:36.247273 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:44:36.247352 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:44:36.247458 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:44:36.247835 (+ 377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:44:36.248839 (+ 1004us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:44:36.251072 (+ 2233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:44:36.252144 (+ 1072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:44:36.252281 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:44:36.252372 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0619 16:44:36.252541 (+ 169us) server_negotiation.cc:299] Negotiation successful
0619 16:44:36.252658 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260619 16:44:36.253464 11186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:44:36.246797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:41455 (local address 127.0.97.2:46353)
0619 16:44:36.247132 (+ 335us) server_negotiation.cc:206] Beginning negotiation
0619 16:44:36.247140 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 16:44:36.247543 (+ 403us) server_negotiation.cc:407] Connection header received
0619 16:44:36.247665 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:44:36.247669 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:44:36.247725 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:44:36.247813 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:44:36.248284 (+ 471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:44:36.249258 (+ 974us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:44:36.251431 (+ 2173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:44:36.252587 (+ 1156us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:44:36.252704 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:44:36.252788 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0619 16:44:36.253124 (+ 336us) server_negotiation.cc:299] Negotiation successful
0619 16:44:36.253253 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260619 16:44:36.253985 11196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:44:36.246639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:46353 (local address 127.0.97.3:41455)
0619 16:44:36.247435 (+ 796us) negotiation.cc:107] Waiting for socket to connect
0619 16:44:36.247450 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 16:44:36.247551 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:44:36.247833 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:44:36.247836 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:44:36.247856 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:44:36.248136 (+ 280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:44:36.248145 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:44:36.249398 (+ 1253us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:44:36.249403 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:44:36.251292 (+ 1889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:44:36.251302 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:44:36.252775 (+ 1473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:44:36.252780 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:44:36.252933 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:44:36.252963 (+ 30us) client_negotiation.cc:769] Sending connection context
0619 16:44:36.253625 (+ 662us) client_negotiation.cc:240] Negotiation successful
0619 16:44:36.253766 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":703,"thread_start_us":545,"threads_started":1}
I20260619 16:44:36.253883 10857 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d53d03e6e3d74e129ae000b1bf4d99c1" candidate_uuid: "2d262a9be62a46429b37d460bb96bd44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fdcb93d084a2480b8bf4f02b5b761d7e" is_pre_election: true
I20260619 16:44:36.254359 10857 raft_consensus.cc:2454] T d53d03e6e3d74e129ae000b1bf4d99c1 P fdcb93d084a2480b8bf4f02b5b761d7e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2d262a9be62a46429b37d460bb96bd44 in term 0.
I20260619 16:44:36.254778 10991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d53d03e6e3d74e129ae000b1bf4d99c1" candidate_uuid: "2d262a9be62a46429b37d460bb96bd44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb6166b6a1fd4de0876f490bb2ccb2a7" is_pre_election: true
I20260619 16:44:36.254983 11060 leader_election.cc:302] T d53d03e6e3d74e129ae000b1bf4d99c1 P 2d262a9be62a46429b37d460bb96bd44 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2d262a9be62a46429b37d460bb96bd44, fdcb93d084a2480b8bf4f02b5b761d7e; no voters:
I20260619 16:44:36.255277 10991 raft_consensus.cc:2454] T d53d03e6e3d74e129ae000b1bf4d99c1 P bb6166b6a1fd4de0876f490bb2ccb2a7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2d262a9be62a46429b37d460bb96bd44 in term 0.
I20260619 16:44:36.255353 11193 raft_consensus.cc:2781] T d53d03e6e3d74e129ae000b1bf4d99c1 P 2d262a9be62a46429b37d460bb96bd44 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 16:44:36.255463 11193 raft_consensus.cc:482] T d53d03e6e3d74e129ae000b1bf4d99c1 P 2d262a9be62a46429b37d460bb96bd44 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 16:44:36.255554 11193 raft_consensus.cc:3037] T d53d03e6e3d74e129ae000b1bf4d99c1 P 2d262a9be62a46429b37d460bb96bd44 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:44:36.257797 11193 raft_consensus.cc:504] T d53d03e6e3d74e129ae000b1bf4d99c1 P 2d262a9be62a46429b37d460bb96bd44 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdcb93d084a2480b8bf4f02b5b761d7e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37931 } } peers { permanent_uuid: "2d262a9be62a46429b37d460bb96bd44" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37847 } } peers { permanent_uuid: "bb6166b6a1fd4de0876f490bb2ccb2a7" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46353 } }
I20260619 16:44:36.258325 11193 leader_election.cc:288] T d53d03e6e3d74e129ae000b1bf4d99c1 P 2d262a9be62a46429b37d460bb96bd44 [CANDIDATE]: Term 1 election: Requested vote from peers fdcb93d084a2480b8bf4f02b5b761d7e (127.0.97.1:37931), bb6166b6a1fd4de0876f490bb2ccb2a7 (127.0.97.2:46353)
I20260619 16:44:36.258668 10857 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d53d03e6e3d74e129ae000b1bf4d99c1" candidate_uuid: "2d262a9be62a46429b37d460bb96bd44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fdcb93d084a2480b8bf4f02b5b761d7e"
I20260619 16:44:36.258767 10991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d53d03e6e3d74e129ae000b1bf4d99c1" candidate_uuid: "2d262a9be62a46429b37d460bb96bd44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb6166b6a1fd4de0876f490bb2ccb2a7"
I20260619 16:44:36.258849 10857 raft_consensus.cc:3037] T d53d03e6e3d74e129ae000b1bf4d99c1 P fdcb93d084a2480b8bf4f02b5b761d7e [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:44:36.258935 10991 raft_consensus.cc:3037] T d53d03e6e3d74e129ae000b1bf4d99c1 P bb6166b6a1fd4de0876f490bb2ccb2a7 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:44:36.260835 10857 raft_consensus.cc:2454] T d53d03e6e3d74e129ae000b1bf4d99c1 P fdcb93d084a2480b8bf4f02b5b761d7e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2d262a9be62a46429b37d460bb96bd44 in term 1.
I20260619 16:44:36.260962 10991 raft_consensus.cc:2454] T d53d03e6e3d74e129ae000b1bf4d99c1 P bb6166b6a1fd4de0876f490bb2ccb2a7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2d262a9be62a46429b37d460bb96bd44 in term 1.
I20260619 16:44:36.261139 11060 leader_election.cc:302] T d53d03e6e3d74e129ae000b1bf4d99c1 P 2d262a9be62a46429b37d460bb96bd44 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2d262a9be62a46429b37d460bb96bd44, fdcb93d084a2480b8bf4f02b5b761d7e; no voters:
I20260619 16:44:36.261381 11193 raft_consensus.cc:2781] T d53d03e6e3d74e129ae000b1bf4d99c1 P 2d262a9be62a46429b37d460bb96bd44 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:44:36.261830 11193 raft_consensus.cc:686] T d53d03e6e3d74e129ae000b1bf4d99c1 P 2d262a9be62a46429b37d460bb96bd44 [term 1 LEADER]: Becoming Leader. State: Replica: 2d262a9be62a46429b37d460bb96bd44, State: Running, Role: LEADER
I20260619 16:44:36.262306 11193 consensus_queue.cc:237] T d53d03e6e3d74e129ae000b1bf4d99c1 P 2d262a9be62a46429b37d460bb96bd44 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdcb93d084a2480b8bf4f02b5b761d7e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37931 } } peers { permanent_uuid: "2d262a9be62a46429b37d460bb96bd44" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37847 } } peers { permanent_uuid: "bb6166b6a1fd4de0876f490bb2ccb2a7" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46353 } }
I20260619 16:44:36.266517 10713 catalog_manager.cc:5495] T d53d03e6e3d74e129ae000b1bf4d99c1 P 2d262a9be62a46429b37d460bb96bd44 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2d262a9be62a46429b37d460bb96bd44 (127.0.97.3). New cstate: current_term: 1 leader_uuid: "2d262a9be62a46429b37d460bb96bd44" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdcb93d084a2480b8bf4f02b5b761d7e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37931 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2d262a9be62a46429b37d460bb96bd44" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37847 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bb6166b6a1fd4de0876f490bb2ccb2a7" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46353 } health_report { overall_health: UNKNOWN } } }
I20260619 16:44:36.296911 11184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:44:36.291949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47684 (local address 127.0.97.3:37847)
0619 16:44:36.292138 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0619 16:44:36.292144 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:44:36.292236 (+ 92us) server_negotiation.cc:407] Connection header received
0619 16:44:36.292429 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:44:36.292433 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:44:36.292515 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:44:36.292637 (+ 122us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 16:44:36.292996 (+ 359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:44:36.293716 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:44:36.294996 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:44:36.295384 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:44:36.295488 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:44:36.295918 (+ 430us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 16:44:36.296389 (+ 471us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 16:44:36.296513 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0619 16:44:36.296600 (+ 87us) server_negotiation.cc:299] Negotiation successful
0619 16:44:36.296682 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260619 16:44:36.303944 10991 raft_consensus.cc:1261] T d53d03e6e3d74e129ae000b1bf4d99c1 P bb6166b6a1fd4de0876f490bb2ccb2a7 [term 1 FOLLOWER]: Refusing update from remote peer 2d262a9be62a46429b37d460bb96bd44: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:44:36.303946 10857 raft_consensus.cc:1261] T d53d03e6e3d74e129ae000b1bf4d99c1 P fdcb93d084a2480b8bf4f02b5b761d7e [term 1 FOLLOWER]: Refusing update from remote peer 2d262a9be62a46429b37d460bb96bd44: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:44:36.304762 11197 consensus_queue.cc:1035] T d53d03e6e3d74e129ae000b1bf4d99c1 P 2d262a9be62a46429b37d460bb96bd44 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb6166b6a1fd4de0876f490bb2ccb2a7" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46353 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:44:36.305040 11193 consensus_queue.cc:1035] T d53d03e6e3d74e129ae000b1bf4d99c1 P 2d262a9be62a46429b37d460bb96bd44 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fdcb93d084a2480b8bf4f02b5b761d7e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37931 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:44:36.326990 10711 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54024:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:44:36.327240 10711 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54024:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:44:36.330422 10711 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e2a0fedb35064355a5c2de8473dae593: Sending DeleteTablet for 3 replicas of tablet d53d03e6e3d74e129ae000b1bf4d99c1
I20260619 16:44:36.331328 10837 tablet_service.cc:1514] Processing DeleteTablet for tablet d53d03e6e3d74e129ae000b1bf4d99c1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:44:36 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:39118
I20260619 16:44:36.331431 10971 tablet_service.cc:1514] Processing DeleteTablet for tablet d53d03e6e3d74e129ae000b1bf4d99c1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:44:36 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:40548
I20260619 16:44:36.331439 11104 tablet_service.cc:1514] Processing DeleteTablet for tablet d53d03e6e3d74e129ae000b1bf4d99c1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:44:36 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:47676
I20260619 16:44:36.333351 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu pid 10774
I20260619 16:44:36.342782 11210 tablet_replica.cc:330] T d53d03e6e3d74e129ae000b1bf4d99c1 P fdcb93d084a2480b8bf4f02b5b761d7e: stopping tablet replica
I20260619 16:44:36.343786 11210 raft_consensus.cc:2229] T d53d03e6e3d74e129ae000b1bf4d99c1 P fdcb93d084a2480b8bf4f02b5b761d7e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 16:44:36.344870 11210 raft_consensus.cc:2258] T d53d03e6e3d74e129ae000b1bf4d99c1 P fdcb93d084a2480b8bf4f02b5b761d7e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:44:36.346223 11211 tablet_replica.cc:330] T d53d03e6e3d74e129ae000b1bf4d99c1 P 2d262a9be62a46429b37d460bb96bd44: stopping tablet replica
I20260619 16:44:36.349210 11211 raft_consensus.cc:2229] T d53d03e6e3d74e129ae000b1bf4d99c1 P 2d262a9be62a46429b37d460bb96bd44 [term 1 LEADER]: Raft consensus shutting down.
I20260619 16:44:36.349901 11211 raft_consensus.cc:2258] T d53d03e6e3d74e129ae000b1bf4d99c1 P 2d262a9be62a46429b37d460bb96bd44 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:44:36.350417 11210 ts_tablet_manager.cc:1905] T d53d03e6e3d74e129ae000b1bf4d99c1 P fdcb93d084a2480b8bf4f02b5b761d7e: Deleting tablet data with delete state TABLET_DATA_DELETED
Jun 19 16:44:36 dist-test-slave-wvz7 krb5kdc[10668](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887474, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260619 16:44:36.356038 11211 ts_tablet_manager.cc:1905] T d53d03e6e3d74e129ae000b1bf4d99c1 P 2d262a9be62a46429b37d460bb96bd44: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:44:36.356669 11212 tablet_replica.cc:330] T d53d03e6e3d74e129ae000b1bf4d99c1 P bb6166b6a1fd4de0876f490bb2ccb2a7: stopping tablet replica
I20260619 16:44:36.356973 11212 raft_consensus.cc:2229] T d53d03e6e3d74e129ae000b1bf4d99c1 P bb6166b6a1fd4de0876f490bb2ccb2a7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 16:44:36.358152 11212 raft_consensus.cc:2258] T d53d03e6e3d74e129ae000b1bf4d99c1 P bb6166b6a1fd4de0876f490bb2ccb2a7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:44:36.359737 11183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:44:36.334440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39130 (local address 127.0.97.1:37931)
0619 16:44:36.334605 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0619 16:44:36.334611 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:44:36.339930 (+ 5319us) server_negotiation.cc:407] Connection header received
0619 16:44:36.340102 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:44:36.340107 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:44:36.340197 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:44:36.340299 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:44:36.344016 (+ 3717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:44:36.345169 (+ 1153us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:44:36.346949 (+ 1780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:44:36.347618 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:44:36.347714 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:44:36.352872 (+ 5158us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:44:36.352884 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:44:36.352900 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:44:36.352943 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:44:36.356510 (+ 3567us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:44:36.357350 (+ 840us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:44:36.357358 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:44:36.357361 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:44:36.357500 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:44:36.357963 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:44:36.357969 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:44:36.357972 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:44:36.358397 (+ 425us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:44:36.358670 (+ 273us) server_negotiation.cc:1035] Waiting for connection context
0619 16:44:36.359189 (+ 519us) server_negotiation.cc:299] Negotiation successful
0619 16:44:36.359391 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260619 16:44:36.360061 11212 ts_tablet_manager.cc:1905] T d53d03e6e3d74e129ae000b1bf4d99c1 P bb6166b6a1fd4de0876f490bb2ccb2a7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:44:36.361302 10897 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:44:36.364014 11211 ts_tablet_manager.cc:1918] T d53d03e6e3d74e129ae000b1bf4d99c1 P 2d262a9be62a46429b37d460bb96bd44: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 16:44:36.364245 11211 log.cc:1196] T d53d03e6e3d74e129ae000b1bf4d99c1 P 2d262a9be62a46429b37d460bb96bd44: Deleting WAL directory at /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887385903558-388-0/minicluster-data/ts-2/wal/wals/d53d03e6e3d74e129ae000b1bf4d99c1
I20260619 16:44:36.364827 11211 ts_tablet_manager.cc:1939] T d53d03e6e3d74e129ae000b1bf4d99c1 P 2d262a9be62a46429b37d460bb96bd44: Deleting consensus metadata
I20260619 16:44:36.366557 10698 catalog_manager.cc:4841] TS 2d262a9be62a46429b37d460bb96bd44 (127.0.97.3:37847): tablet d53d03e6e3d74e129ae000b1bf4d99c1 (table test-table [id=504e126c62464b889bbbf81d27ecaf3b]) successfully deleted
I20260619 16:44:36.367409 11212 ts_tablet_manager.cc:1918] T d53d03e6e3d74e129ae000b1bf4d99c1 P bb6166b6a1fd4de0876f490bb2ccb2a7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 16:44:36.367581 11212 log.cc:1196] T d53d03e6e3d74e129ae000b1bf4d99c1 P bb6166b6a1fd4de0876f490bb2ccb2a7: Deleting WAL directory at /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887385903558-388-0/minicluster-data/ts-1/wal/wals/d53d03e6e3d74e129ae000b1bf4d99c1
I20260619 16:44:36.368248 11212 ts_tablet_manager.cc:1939] T d53d03e6e3d74e129ae000b1bf4d99c1 P bb6166b6a1fd4de0876f490bb2ccb2a7: Deleting consensus metadata
I20260619 16:44:36.370411 10698 catalog_manager.cc:4841] TS bb6166b6a1fd4de0876f490bb2ccb2a7 (127.0.97.2:46353): tablet d53d03e6e3d74e129ae000b1bf4d99c1 (table test-table [id=504e126c62464b889bbbf81d27ecaf3b]) successfully deleted
I20260619 16:44:36.452042 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu with pid 10774
W20260619 16:44:36.472733 10700 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.97.1:37931 (error 108)
I20260619 16:44:36.473395 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu pid 10908
I20260619 16:44:36.474812 11181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:44:36.474312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:37931 (local address 127.0.0.1:39134)
0619 16:44:36.474528 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0619 16:44:36.474647 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:37931: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120}
W20260619 16:44:36.475135 10700 catalog_manager.cc:4584] TS fdcb93d084a2480b8bf4f02b5b761d7e (127.0.97.1:37931): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d53d03e6e3d74e129ae000b1bf4d99c1: Network error: Client connection negotiation failed: client connection to 127.0.97.1:37931: connect: Connection refused (error 111)
Jun 19 16:44:36 dist-test-slave-wvz7 krb5kdc[10668](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887474, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260619 16:44:36.490442 11186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:44:36.474259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40564 (local address 127.0.97.2:46353)
0619 16:44:36.474443 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0619 16:44:36.474449 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:44:36.474508 (+ 59us) server_negotiation.cc:407] Connection header received
0619 16:44:36.474717 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:44:36.474722 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:44:36.474804 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:44:36.474909 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:44:36.476437 (+ 1528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:44:36.477158 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:44:36.479072 (+ 1914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:44:36.479476 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:44:36.479577 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:44:36.484107 (+ 4530us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:44:36.484125 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:44:36.484144 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:44:36.484188 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:44:36.487705 (+ 3517us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:44:36.488349 (+ 644us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:44:36.488358 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:44:36.488362 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:44:36.488510 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:44:36.488823 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:44:36.488827 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:44:36.488831 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:44:36.489310 (+ 479us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:44:36.489598 (+ 288us) server_negotiation.cc:1035] Waiting for connection context
0619 16:44:36.489841 (+ 243us) server_negotiation.cc:299] Negotiation successful
0619 16:44:36.490076 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260619 16:44:36.491259 11031 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:44:36.528940 11181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:44:36.528406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:37931 (local address 127.0.0.1:39138)
0619 16:44:36.528652 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0619 16:44:36.528765 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:37931: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99}
I20260619 16:44:36.530082 11181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:44:36.529642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:37931 (local address 127.0.0.1:39152)
0619 16:44:36.529825 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0619 16:44:36.529933 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:37931: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260619 16:44:36.578639 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu with pid 10908
I20260619 16:44:36.590758 11181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:44:36.590218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:37931 (local address 127.0.0.1:39154)
0619 16:44:36.590455 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0619 16:44:36.590564 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:37931: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260619 16:44:36.591882 11181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:44:36.591495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:37931 (local address 127.0.0.1:39168)
0619 16:44:36.591654 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0619 16:44:36.591751 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:37931: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260619 16:44:36.594502 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu pid 11041
Jun 19 16:44:36 dist-test-slave-wvz7 krb5kdc[10668](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887474, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260619 16:44:36.610286 11184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:44:36.595228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47700 (local address 127.0.97.3:37847)
0619 16:44:36.595402 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0619 16:44:36.595408 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:44:36.595425 (+ 17us) server_negotiation.cc:407] Connection header received
0619 16:44:36.595486 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:44:36.595490 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:44:36.595565 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:44:36.595647 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:44:36.596826 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:44:36.597636 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:44:36.598805 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:44:36.599293 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:44:36.599402 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:44:36.604205 (+ 4803us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:44:36.604217 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:44:36.604232 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:44:36.604276 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:44:36.607334 (+ 3058us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:44:36.607988 (+ 654us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:44:36.607995 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:44:36.607997 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:44:36.608161 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:44:36.608556 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:44:36.608561 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:44:36.608563 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:44:36.608936 (+ 373us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:44:36.609260 (+ 324us) server_negotiation.cc:1035] Waiting for connection context
0619 16:44:36.609566 (+ 306us) server_negotiation.cc:299] Negotiation successful
0619 16:44:36.609899 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260619 16:44:36.611042 11164 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:44:36.672361 11181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:44:36.671799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:37931 (local address 127.0.0.1:39182)
0619 16:44:36.672059 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0619 16:44:36.672174 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:37931: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260619 16:44:36.673488 11181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:44:36.673036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:37931 (local address 127.0.0.1:39188)
0619 16:44:36.673228 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0619 16:44:36.673333 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:37931: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20260619 16:44:36.728206 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu with pid 11041
I20260619 16:44:36.742811 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu pid 10681
I20260619 16:44:36.743870 10743 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:44:36.847599 11181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:44:36.847076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:37931 (local address 127.0.0.1:39200)
0619 16:44:36.847268 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0619 16:44:36.847349 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:37931: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260619 16:44:36.847921 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu with pid 10681
2026-06-19T16:44:36Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (3811 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887385903558-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 16:44:36 dist-test-slave-wvz7 krb5kdc[11221](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 16:44:36 dist-test-slave-wvz7 krb5kdc[11221](info): set up 2 sockets
Jun 19 16:44:36 dist-test-slave-wvz7 krb5kdc[11221](info): commencing operation
krb5kdc: starting...
W20260619 16:44:37.854480 388 mini_kdc.cc:121] Time spent starting KDC: real 0.970s user 0.002s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887385903558-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887385903558-388-0/krb5kdc/test-user.keytab.
Jun 19 16:44:37 dist-test-slave-wvz7 krb5kdc[11221](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887477, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T16:44:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T16:44:38Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887385903558-388-0/krb5kdc/rangeradmin_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887385903558-388-0/krb5kdc/rangeradmin_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887385903558-388-0/krb5kdc/rangerlookup_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887385903558-388-0/krb5kdc/rangerlookup_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887385903558-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887385903558-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
I20260619 16:44:38.126058 388 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887385903558-388-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskEs2EH8/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887385903558-388-0/minicluster-data/postgres -l logfile start
2026-06-19 16:44:40.402 UTC [11251] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-06-19 16:44:40.402 UTC [11251] LOG: listening on IPv4 address "127.0.97.20", port 36721
2026-06-19 16:44:40.405 UTC [11251] LOG: listening on Unix socket "/tmp/.s.PGSQL.36721"
2026-06-19 16:44:40.426 UTC [11254] LOG: database system was shut down at 2026-06-19 16:44:39 UTC
2026-06-19 16:44:40.429 UTC [11251] LOG: database system is ready to accept connections
I20260619 16:44:41.309983 388 mini_postgres.cc:96] Postgres bound to 36721
2026-06-19 16:44:41.325 UTC [11262] FATAL: database "slave" does not exist
127.0.97.20:36721 - accepting connections
I20260619 16:44:41.325830 388 mini_ranger.cc:161] Starting Ranger...
I20260619 16:44:41.348268 388 mini_ranger.cc:84] Created miniranger Postgres user
I20260619 16:44:41.795440 388 mini_ranger.cc:87] Created ranger Postgres database
I20260619 16:44:41.795570 388 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887385903558-388-0/minicluster-data/ranger-admin
2026-06-19 16:44:42,071 [I] DB FLAVOR :POSTGRES
2026-06-19 16:44:42,071 [I] --------- Verifying Ranger DB connection ---------
2026-06-19 16:44:42,071 [I] Checking connection..
2026-06-19 16:44:42,071 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskEs2EH8/build/asan/bin/postgresql.jar:/tmp/dist-test-taskEs2EH8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:36721/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-06-19 16:44:42,325 [I] Checking connection passed.
2026-06-19 16:44:42,325 [I] --------- Verifying version history table ---------
2026-06-19 16:44:42,325 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskEs2EH8/build/asan/bin/postgresql.jar:/tmp/dist-test-taskEs2EH8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:36721/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-19 16:44:42,566 [I] Table x_db_version_h does not exist in database ranger
2026-06-19 16:44:42,566 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskEs2EH8/build/asan/bin/postgresql.jar:/tmp/dist-test-taskEs2EH8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:36721/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-19 16:44:42,808 [I] Table x_db_version_h does not exist in database ranger
2026-06-19 16:44:42,808 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-06-19 16:44:42,808 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskEs2EH8/build/asan/bin/postgresql.jar:/tmp/dist-test-taskEs2EH8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:36721/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskEs2EH8/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-06-19 16:44:43.054 UTC [11326] WARNING: there is no transaction in progress
2026-06-19 16:44:43,060 [I] create_dbversion_catalog.sql file imported successfully
2026-06-19 16:44:43,060 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskEs2EH8/build/asan/bin/postgresql.jar:/tmp/dist-test-taskEs2EH8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:36721/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-19 16:44:43,297 [I] Table x_db_version_h already exists in database 'ranger'
2026-06-19 16:44:43,297 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-19 16:44:43,298 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskEs2EH8/build/asan/bin/postgresql.jar:/tmp/dist-test-taskEs2EH8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:36721/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-06-19 16:44:43,544 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskEs2EH8/build/asan/bin/postgresql.jar:/tmp/dist-test-taskEs2EH8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:36721/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-06-19 16:44:43,772 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskEs2EH8/build/asan/bin/postgresql.jar:/tmp/dist-test-taskEs2EH8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:36721/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-wvz7.c.gcp-upstream.internal','N') ;"
2026-06-19 16:44:44,009 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskEs2EH8/build/asan/bin/postgresql.jar:/tmp/dist-test-taskEs2EH8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:36721/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-19 16:44:44,229 [I] Table x_portal_user does not exist in database ranger
2026-06-19 16:44:44,229 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskEs2EH8/build/asan/bin/postgresql.jar:/tmp/dist-test-taskEs2EH8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:36721/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-19 16:44:44,477 [I] Table x_policy_ref_group does not exist in database ranger
2026-06-19 16:44:44,478 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskEs2EH8/build/asan/bin/postgresql.jar:/tmp/dist-test-taskEs2EH8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:36721/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-19 16:44:44,714 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskEs2EH8/build/asan/bin/postgresql.jar:/tmp/dist-test-taskEs2EH8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:36721/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-19 16:44:44,934 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-06-19 16:44:44,934 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskEs2EH8/build/asan/bin/postgresql.jar:/tmp/dist-test-taskEs2EH8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:36721/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskEs2EH8/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-06-19 16:44:45.818 UTC [11434] WARNING: there is no transaction in progress
2026-06-19 16:44:45.833 UTC [11434] WARNING: there is no transaction in progress
2026-06-19 16:44:45.847 UTC [11434] WARNING: there is no transaction in progress
2026-06-19 16:44:45.862 UTC [11434] WARNING: there is no transaction in progress
2026-06-19 16:44:45.877 UTC [11434] WARNING: there is no transaction in progress
2026-06-19 16:44:45.894 UTC [11434] WARNING: there is no transaction in progress
2026-06-19 16:44:45.942 UTC [11434] WARNING: there is no transaction in progress
2026-06-19 16:44:45.948 UTC [11434] WARNING: there is no transaction in progress
2026-06-19 16:44:45.961 UTC [11434] WARNING: there is no transaction in progress
2026-06-19 16:44:45.970 UTC [11434] WARNING: there is no transaction in progress
2026-06-19 16:44:45.979 UTC [11434] WARNING: there is no transaction in progress
2026-06-19 16:44:45.992 UTC [11434] WARNING: there is no transaction in progress
2026-06-19 16:44:46.001 UTC [11434] WARNING: there is no transaction in progress
2026-06-19 16:44:46.007 UTC [11434] WARNING: there is no transaction in progress
2026-06-19 16:44:46.012 UTC [11434] WARNING: there is no transaction in progress
2026-06-19 16:44:46.018 UTC [11434] WARNING: there is no transaction in progress
2026-06-19 16:44:46,587 [I] ranger_core_db_postgres.sql file imported successfully
2026-06-19 16:44:46,588 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskEs2EH8/build/asan/bin/postgresql.jar:/tmp/dist-test-taskEs2EH8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:36721/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-06-19 16:44:46,814 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-06-19 16:44:46,814 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskEs2EH8/build/asan/bin/postgresql.jar:/tmp/dist-test-taskEs2EH8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:36721/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-19 16:44:47,056 [I] Table x_portal_user already exists in database 'ranger'
2026-06-19 16:44:47,056 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskEs2EH8/build/asan/bin/postgresql.jar:/tmp/dist-test-taskEs2EH8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:36721/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-19 16:44:47,304 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-06-19 16:44:47,305 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskEs2EH8/build/asan/bin/postgresql.jar:/tmp/dist-test-taskEs2EH8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:36721/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-19 16:44:47,527 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskEs2EH8/build/asan/bin/postgresql.jar:/tmp/dist-test-taskEs2EH8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:36721/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-19 16:44:47,752 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskEs2EH8/build/asan/bin/postgresql.jar:/tmp/dist-test-taskEs2EH8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:36721/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-wvz7.c.gcp-upstream.internal';"
2026-06-19 16:44:47,985 [I] CORE_DB_SCHEMA import status has been updated
2026-06-19 16:44:47,985 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskEs2EH8/build/asan/bin/postgresql.jar:/tmp/dist-test-taskEs2EH8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:36721/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-06-19 16:44:48,232 [I] DB_PATCHES have already been applied
I20260619 16:44:48.237496 388 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887385903558-388-0/minicluster-data/ranger-admin:/tmp/dist-test-taskEs2EH8/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskEs2EH8/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskEs2EH8/build/asan/bin/postgresql.jar:/tmp/dist-test-taskEs2EH8/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260619 16:44:48.237579 388 mini_ranger.cc:193] Using host: 127.0.97.20
I20260619 16:44:48.250607 388 mini_ranger.cc:227] Ranger admin URL: http://127.0.97.20:37207
Jun 19, 2026 4:44:48 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 19, 2026 4:44:48 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 19, 2026 4:44:48 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887385903558-388-0/minicluster-data/ranger-admin/ews/webapp
Jun 19, 2026 4:44:48 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887385903558-388-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Jun 19, 2026 4:44:48 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887385903558-388-0/minicluster-data/ranger-admin/ews/webapp] .....
Jun 19, 2026 4:44:49 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887385903558-388-0/minicluster-data/ranger-admin/ews/webapp].
Jun 19, 2026 4:44:49 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260619 16:44:49.271400 388 mini_ranger.cc:160] Time spent starting Ranger: real 7.946s user 0.000s sys 0.034s
Jun 19, 2026 4:45:09 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Jun 19, 2026 4:45:09 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.XKeyREST
Jun 19, 2026 4:45:09 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Jun 19, 2026 4:45:09 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Jun 19, 2026 4:45:09 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Jun 19, 2026 4:45:09 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Jun 19, 2026 4:45:09 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Jun 19, 2026 4:45:09 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Jun 19, 2026 4:45:09 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Jun 19, 2026 4:45:09 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Jun 19, 2026 4:45:09 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Jun 19, 2026 4:45:09 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Jun 19, 2026 4:45:09 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Jun 19, 2026 4:45:09 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Jun 19, 2026 4:45:09 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Jun 19, 2026 4:45:09 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Jun 19, 2026 4:45:09 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Jun 19, 2026 4:45:10 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260619 16:45:10.548017 388 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887385903558-388-0/krb5kdc/rangerkms_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887385903558-388-0/krb5kdc/rangerkms_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.97.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.97.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887385903558-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887385903558-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Jun 19 16:45:10 dist-test-slave-wvz7 krb5kdc[11221](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887510, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260619 16:45:10.692763 388 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260619 16:45:10.710032 388 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260619 16:45:11.476470 388 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260619 16:45:11.476637 388 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887385903558-388-0/minicluster-data/ranger-kms
I20260619 16:45:11.476733 388 mini_ranger_kms.cc:227] Using postgres at 127.0.97.20:36721
2026-06-19 16:45:11,855 [I] DB FLAVOR :POSTGRES
2026-06-19 16:45:11,855 [I] --------- Verifying Ranger DB connection ---------
2026-06-19 16:45:11,855 [I] Checking connection
2026-06-19 16:45:11,855 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskEs2EH8/build/asan/bin/postgresql.jar:/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887385903558-388-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:36721/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-06-19 16:45:12,094 [I] connection success
2026-06-19 16:45:12,094 [I] --------- Verifying Ranger DB tables ---------
2026-06-19 16:45:12,094 [I] Verifying table ranger_masterkey in database rangerkms
2026-06-19 16:45:12,094 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskEs2EH8/build/asan/bin/postgresql.jar:/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887385903558-388-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:36721/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-06-19 16:45:12,317 [I] Table ranger_masterkey does not exist in database rangerkms
2026-06-19 16:45:12,317 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-19 16:45:12,317 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-06-19 16:45:12,317 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskEs2EH8/build/asan/bin/postgresql.jar:/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887385903558-388-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:36721/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887385903558-388-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-06-19 16:45:12,561 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260619 16:45:13.027588 388 mini_ranger_kms.cc:310] Created kms service
I20260619 16:45:13.237151 388 mini_ranger_kms.cc:326] Created kudu user
I20260619 16:45:13.280743 388 mini_ranger_kms.cc:343] Created rangerkms user
Jun 19 16:45:13 dist-test-slave-wvz7 krb5kdc[11221](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887510, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260619 16:45:13.669891 388 mini_ranger_kms.cc:384] Added ranger policy
I20260619 16:45:13.670166 388 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887385903558-388-0/minicluster-data/ranger-kms:/tmp/dist-test-taskEs2EH8/build/asan/bin/postgresql.jar:/tmp/dist-test-taskEs2EH8/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskEs2EH8/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260619 16:45:13.670215 388 mini_ranger_kms.cc:240] Using host: 127.0.97.20
I20260619 16:45:13.689393 388 mini_ranger_kms.cc:276] Ranger KMS PID: 11614
I20260619 16:45:13.689530 388 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.97.20:38923
Jun 19, 2026 4:45:14 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 19, 2026 4:45:14 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 19, 2026 4:45:14 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887385903558-388-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Jun 19, 2026 4:45:14 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887385903558-388-0/minicluster-data/ranger-kms/ews/webapp] .....
Jun 19, 2026 4:45:15 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887385903558-388-0/minicluster-data/ranger-kms/ews/webapp].
Jun 19, 2026 4:45:15 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Jun 19 16:45:15 dist-test-slave-wvz7 krb5kdc[11221](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781887515, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.97.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:45:15.748016 388 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.055s user 0.101s sys 0.290s
I20260619 16:45:15.748235 388 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260619 16:45:15.748415 388 mini_ranger_kms.cc:401] 127.0.97.20:38923/kms/v1/keys
Jun 19 16:45:16 dist-test-slave-wvz7 krb5kdc[11221](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887515, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.97.20@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
Jun 19 16:45:16 dist-test-slave-wvz7 krb5kdc[11221](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887515, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.97.20@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
Jun 19 16:45:18 dist-test-slave-wvz7 krb5kdc[11221](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887518, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887385903558-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887385903558-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260619 16:45:18.922676 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu
/tmp/dist-test-taskEs2EH8/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887385903558-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887385903558-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887385903558-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887385903558-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37823
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887385903558-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42629
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:37823
--ranger_config_path=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887385903558-388-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:38923/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887385903558-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887385903558-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887385903558-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:45:19.239064 11665 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:45:19.239328 11665 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:45:19.239454 11665 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:45:19.246372 11665 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:45:19.246888 11665 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:45:19.246966 11665 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:45:19.247045 11665 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:45:19.247097 11665 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:45:19.256798 11665 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42629
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887385903558-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887385903558-388-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.97.20:38923/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:37823
--ranger_config_path=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887385903558-388-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887385903558-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37823
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887385903558-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887385903558-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:45:19.258101 11665 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:45:19.259341 11665 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:45:19.270527 11670 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:45:19.270445 11671 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:45:19.270501 11673 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:45:19.271389 11665 server_base.cc:1034] running on GCE node
I20260619 16:45:19.272089 11665 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:45:19.273440 11665 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:45:19.274670 11665 hybrid_clock.cc:648] HybridClock initialized: now 1781887519274626 us; error 37 us; skew 500 ppm
I20260619 16:45:19.275017 11665 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:45:19 dist-test-slave-wvz7 krb5kdc[11221](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887519, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:45:19.278249 11665 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260619 16:45:19.279758 11665 webserver.cc:460] Webserver started at http://127.0.97.62:37879/ using document root <none> and password file <none>
I20260619 16:45:19.280401 11665 fs_manager.cc:359] Metadata directory not provided
I20260619 16:45:19.280506 11665 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:45:19.280778 11665 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 16:45:19 dist-test-slave-wvz7 krb5kdc[11221](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887519, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260619 16:45:19.338886 11665 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887385903558-388-0/minicluster-data/master-0/data/instance:
uuid: "cec89906d5fc4e5fa81d027ada4f6753"
format_stamp: "Formatted at 2026-06-19 16:45:19 on dist-test-slave-wvz7"
server_key: "9de1c4b139fa899b02fb8b1421aba099"
server_key_iv: "5e2b3d34a2d0f90e02fadc2a2b407b82"
server_key_version: "kuduclusterkey@0"
I20260619 16:45:19.339745 11665 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887385903558-388-0/minicluster-data/master-0/wal/instance:
uuid: "cec89906d5fc4e5fa81d027ada4f6753"
format_stamp: "Formatted at 2026-06-19 16:45:19 on dist-test-slave-wvz7"
server_key: "9de1c4b139fa899b02fb8b1421aba099"
server_key_iv: "5e2b3d34a2d0f90e02fadc2a2b407b82"
server_key_version: "kuduclusterkey@0"
I20260619 16:45:19.345180 11665 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260619 16:45:19.361413 11680 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:45:19.362972 11665 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260619 16:45:19.363154 11665 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887385903558-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887385903558-388-0/minicluster-data/master-0/wal
uuid: "cec89906d5fc4e5fa81d027ada4f6753"
format_stamp: "Formatted at 2026-06-19 16:45:19 on dist-test-slave-wvz7"
server_key: "9de1c4b139fa899b02fb8b1421aba099"
server_key_iv: "5e2b3d34a2d0f90e02fadc2a2b407b82"
server_key_version: "kuduclusterkey@0"
I20260619 16:45:19.363374 11665 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887385903558-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887385903558-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887385903558-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:45:19.380748 11665 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:45:19.385002 11665 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:45:19.385263 11665 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:45:19.404223 11665 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:37823
I20260619 16:45:19.404246 11731 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:37823 every 8 connection(s)
I20260619 16:45:19.405553 11665 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887385903558-388-0/minicluster-data/master-0/data/info.pb
I20260619 16:45:19.406296 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu as pid 11665
I20260619 16:45:19.406508 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887385903558-388-0/minicluster-data/master-0/wal/instance
I20260619 16:45:19.407351 388 external_mini_cluster.cc:1428] Setting key b7cbee9b13d0a3b128d1a13e0b818ab3
I20260619 16:45:19.410137 11732 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:45:19.424278 11732 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cec89906d5fc4e5fa81d027ada4f6753: Bootstrap starting.
I20260619 16:45:19.427923 11732 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cec89906d5fc4e5fa81d027ada4f6753: Neither blocks nor log segments found. Creating new log.
I20260619 16:45:19.429067 11732 log.cc:824] T 00000000000000000000000000000000 P cec89906d5fc4e5fa81d027ada4f6753: Log is configured to *not* fsync() on all Append() calls
Jun 19 16:45:19 dist-test-slave-wvz7 krb5kdc[11221](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887518, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 16:45:19.431849 11732 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cec89906d5fc4e5fa81d027ada4f6753: No bootstrap required, opened a new log
I20260619 16:45:19.437528 11735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:45:19.415414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55008 (local address 127.0.97.62:37823)
0619 16:45:19.416307 (+ 893us) server_negotiation.cc:206] Beginning negotiation
0619 16:45:19.416323 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0619 16:45:19.418323 (+ 2000us) server_negotiation.cc:407] Connection header received
0619 16:45:19.420024 (+ 1701us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:45:19.420045 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:45:19.420457 (+ 412us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:45:19.420786 (+ 329us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:45:19.422330 (+ 1544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:45:19.423598 (+ 1268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:45:19.425206 (+ 1608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:45:19.425917 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:45:19.426092 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:45:19.430338 (+ 4246us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:45:19.430346 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:45:19.430360 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:45:19.430393 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:45:19.433177 (+ 2784us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:45:19.433976 (+ 799us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:45:19.433982 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:45:19.433989 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:45:19.434155 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:45:19.434518 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:45:19.434522 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:45:19.434524 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:45:19.435034 (+ 510us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:45:19.435282 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0619 16:45:19.436180 (+ 898us) server_negotiation.cc:299] Negotiation successful
0619 16:45:19.436529 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":635,"thread_start_us":381,"threads_started":1}
I20260619 16:45:19.440109 11732 raft_consensus.cc:348] T 00000000000000000000000000000000 P cec89906d5fc4e5fa81d027ada4f6753 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cec89906d5fc4e5fa81d027ada4f6753" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37823 } }
I20260619 16:45:19.440434 11732 raft_consensus.cc:374] T 00000000000000000000000000000000 P cec89906d5fc4e5fa81d027ada4f6753 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:45:19.440517 11732 raft_consensus.cc:729] T 00000000000000000000000000000000 P cec89906d5fc4e5fa81d027ada4f6753 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cec89906d5fc4e5fa81d027ada4f6753, State: Initialized, Role: FOLLOWER
I20260619 16:45:19.441025 11732 consensus_queue.cc:260] T 00000000000000000000000000000000 P cec89906d5fc4e5fa81d027ada4f6753 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cec89906d5fc4e5fa81d027ada4f6753" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37823 } }
I20260619 16:45:19.441234 11732 raft_consensus.cc:388] T 00000000000000000000000000000000 P cec89906d5fc4e5fa81d027ada4f6753 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:45:19.441354 11732 raft_consensus.cc:482] T 00000000000000000000000000000000 P cec89906d5fc4e5fa81d027ada4f6753 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:45:19.441485 11732 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cec89906d5fc4e5fa81d027ada4f6753 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:45:19.443477 11732 raft_consensus.cc:504] T 00000000000000000000000000000000 P cec89906d5fc4e5fa81d027ada4f6753 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cec89906d5fc4e5fa81d027ada4f6753" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37823 } }
I20260619 16:45:19.443929 11732 leader_election.cc:302] T 00000000000000000000000000000000 P cec89906d5fc4e5fa81d027ada4f6753 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cec89906d5fc4e5fa81d027ada4f6753; no voters:
I20260619 16:45:19.444453 11732 leader_election.cc:288] T 00000000000000000000000000000000 P cec89906d5fc4e5fa81d027ada4f6753 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 16:45:19.444571 11737 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cec89906d5fc4e5fa81d027ada4f6753 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:45:19.445163 11737 raft_consensus.cc:686] T 00000000000000000000000000000000 P cec89906d5fc4e5fa81d027ada4f6753 [term 1 LEADER]: Becoming Leader. State: Replica: cec89906d5fc4e5fa81d027ada4f6753, State: Running, Role: LEADER
I20260619 16:45:19.445681 11737 consensus_queue.cc:237] T 00000000000000000000000000000000 P cec89906d5fc4e5fa81d027ada4f6753 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cec89906d5fc4e5fa81d027ada4f6753" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37823 } }
I20260619 16:45:19.446491 11732 sys_catalog.cc:564] T 00000000000000000000000000000000 P cec89906d5fc4e5fa81d027ada4f6753 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:45:19.448809 11739 sys_catalog.cc:455] T 00000000000000000000000000000000 P cec89906d5fc4e5fa81d027ada4f6753 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cec89906d5fc4e5fa81d027ada4f6753. Latest consensus state: current_term: 1 leader_uuid: "cec89906d5fc4e5fa81d027ada4f6753" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cec89906d5fc4e5fa81d027ada4f6753" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37823 } } }
I20260619 16:45:19.448845 11738 sys_catalog.cc:455] T 00000000000000000000000000000000 P cec89906d5fc4e5fa81d027ada4f6753 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cec89906d5fc4e5fa81d027ada4f6753" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cec89906d5fc4e5fa81d027ada4f6753" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37823 } } }
I20260619 16:45:19.449204 11739 sys_catalog.cc:458] T 00000000000000000000000000000000 P cec89906d5fc4e5fa81d027ada4f6753 [sys.catalog]: This master's current role is: LEADER
I20260619 16:45:19.449214 11738 sys_catalog.cc:458] T 00000000000000000000000000000000 P cec89906d5fc4e5fa81d027ada4f6753 [sys.catalog]: This master's current role is: LEADER
I20260619 16:45:19.450903 11744 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:45:19.455338 11744 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:45:19.458739 11732 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887385903558-388-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260619 16:45:19.472838 11744 catalog_manager.cc:1347] Generated new cluster ID: ae7c89b6d7cf49e39c26ed27e965b071
I20260619 16:45:19.472983 11744 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:45:19.491333 11744 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 16:45:19.493175 11744 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:45:19.503604 11744 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cec89906d5fc4e5fa81d027ada4f6753: Generated new TSK 0
I20260619 16:45:19.504741 11744 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887385903558-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887385903558-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260619 16:45:20.221011 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu
/tmp/dist-test-taskEs2EH8/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887385903558-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887385903558-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887385903558-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887385903558-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:38923/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37823
--keytab_file=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887385903558-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42629
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887385903558-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887385903558-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887385903558-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 16:45:20 dist-test-slave-wvz7 krb5kdc[11221](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781887520, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260619 16:45:20.601142 11777 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:45:20.601540 11777 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:45:20.601764 11777 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:45:20.610716 11777 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:45:20.610816 11777 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:45:20.610952 11777 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260619 16:45:20.622491 11777 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42629
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887385903558-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887385903558-388-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.97.20:38923/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887385903558-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887385903558-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37823
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887385903558-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:45:20.623888 11777 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:45:20.625162 11777 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:45:20.643105 11787 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:45:20.646459 11788 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:45:20.654630 11790 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:45:20.654906 11777 server_base.cc:1034] running on GCE node
I20260619 16:45:20.655694 11777 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:45:20.659437 11777 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:45:20.660872 11777 hybrid_clock.cc:648] HybridClock initialized: now 1781887520660805 us; error 56 us; skew 500 ppm
I20260619 16:45:20.661305 11777 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:45:20 dist-test-slave-wvz7 krb5kdc[11221](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887520, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:45:20.665479 11777 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260619 16:45:20.667522 11777 webserver.cc:460] Webserver started at http://127.0.97.1:44417/ using document root <none> and password file <none>
I20260619 16:45:20.668423 11777 fs_manager.cc:359] Metadata directory not provided
I20260619 16:45:20.668550 11777 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:45:20.668926 11777 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 16:45:20 dist-test-slave-wvz7 krb5kdc[11221](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887520, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260619 16:45:20.699067 11777 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887385903558-388-0/minicluster-data/ts-0/data/instance:
uuid: "1616c37cb2f54a8f9845332f263f9816"
format_stamp: "Formatted at 2026-06-19 16:45:20 on dist-test-slave-wvz7"
server_key: "4fdff0a927977fd0a58cfedd134a8d89"
server_key_iv: "b53daedf6ca02b0ad51ef3ff3e76e7fa"
server_key_version: "kuduclusterkey@0"
I20260619 16:45:20.700338 11777 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887385903558-388-0/minicluster-data/ts-0/wal/instance:
uuid: "1616c37cb2f54a8f9845332f263f9816"
format_stamp: "Formatted at 2026-06-19 16:45:20 on dist-test-slave-wvz7"
server_key: "4fdff0a927977fd0a58cfedd134a8d89"
server_key_iv: "b53daedf6ca02b0ad51ef3ff3e76e7fa"
server_key_version: "kuduclusterkey@0"
I20260619 16:45:20.707376 11777 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.001s
I20260619 16:45:20.746435 11797 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:45:20.748494 11777 fs_manager.cc:725] Time spent opening block manager: real 0.019s user 0.009s sys 0.008s
I20260619 16:45:20.748709 11777 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887385903558-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887385903558-388-0/minicluster-data/ts-0/wal
uuid: "1616c37cb2f54a8f9845332f263f9816"
format_stamp: "Formatted at 2026-06-19 16:45:20 on dist-test-slave-wvz7"
server_key: "4fdff0a927977fd0a58cfedd134a8d89"
server_key_iv: "b53daedf6ca02b0ad51ef3ff3e76e7fa"
server_key_version: "kuduclusterkey@0"
I20260619 16:45:20.748950 11777 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887385903558-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887385903558-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887385903558-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:45:20.770582 11777 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:45:20.775759 11777 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:45:20.776091 11777 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:45:20.777498 11777 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:45:20.779309 11777 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:45:20.779405 11777 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:45:20.779577 11777 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:45:20.779675 11777 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 16:45:20 dist-test-slave-wvz7 krb5kdc[11221](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887520, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260619 16:45:21.190897 11777 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:35635
I20260619 16:45:21.190912 11909 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:35635 every 8 connection(s)
I20260619 16:45:21.192214 11777 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887385903558-388-0/minicluster-data/ts-0/data/info.pb
I20260619 16:45:21.200111 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu as pid 11777
I20260619 16:45:21.200328 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887385903558-388-0/minicluster-data/ts-0/wal/instance
I20260619 16:45:21.201128 388 external_mini_cluster.cc:1428] Setting key 65f5da830dbd55fa8fa6d4f73960a7a3
Jun 19 16:45:21 dist-test-slave-wvz7 krb5kdc[11221](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887520, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 16:45:21.241674 11913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:45:21.195678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37823 (local address 127.0.97.1:45639)
0619 16:45:21.208199 (+ 12521us) negotiation.cc:107] Waiting for socket to connect
0619 16:45:21.208272 (+ 73us) client_negotiation.cc:174] Beginning negotiation
0619 16:45:21.210898 (+ 2626us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:45:21.215439 (+ 4541us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:45:21.215452 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:45:21.216032 (+ 580us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:45:21.216823 (+ 791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:45:21.216849 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:45:21.218484 (+ 1635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:45:21.218489 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:45:21.220363 (+ 1874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:45:21.220377 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:45:21.221499 (+ 1122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:45:21.221505 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:45:21.221790 (+ 285us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:45:21.227380 (+ 5590us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:45:21.227411 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:45:21.230508 (+ 3097us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:45:21.236992 (+ 6484us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:45:21.237002 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:45:21.237021 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:45:21.237593 (+ 572us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:45:21.238032 (+ 439us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:45:21.238037 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:45:21.238042 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:45:21.238270 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:45:21.238964 (+ 694us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:45:21.238973 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:45:21.239400 (+ 427us) client_negotiation.cc:769] Sending connection context
0619 16:45:21.239725 (+ 325us) client_negotiation.cc:240] Negotiation successful
0619 16:45:21.240257 (+ 532us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12149,"thread_start_us":15551,"threads_started":1}
I20260619 16:45:21.242480 11912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:45:21.194968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:45639 (local address 127.0.97.62:37823)
0619 16:45:21.214086 (+ 19118us) server_negotiation.cc:206] Beginning negotiation
0619 16:45:21.214092 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:45:21.214114 (+ 22us) server_negotiation.cc:407] Connection header received
0619 16:45:21.214187 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:45:21.214193 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:45:21.214305 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:45:21.217112 (+ 2807us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:45:21.217276 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:45:21.218316 (+ 1040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:45:21.220522 (+ 2206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:45:21.221113 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:45:21.221234 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:45:21.230772 (+ 9538us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:45:21.230778 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:45:21.230784 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:45:21.230822 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:45:21.236819 (+ 5997us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:45:21.237771 (+ 952us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:45:21.237777 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:45:21.237780 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:45:21.237912 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:45:21.238403 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:45:21.238408 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:45:21.238411 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:45:21.238753 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:45:21.238879 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0619 16:45:21.241902 (+ 3023us) server_negotiation.cc:299] Negotiation successful
0619 16:45:21.242126 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18903,"thread_start_us":18927,"threads_started":1}
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260619 16:45:21.244009 11910 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37823
I20260619 16:45:21.244342 11910 heartbeater.cc:464] Registering TS with master...
I20260619 16:45:21.245079 11910 heartbeater.cc:511] Master 127.0.97.62:37823 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260619 16:45:21.247545 11697 ts_manager.cc:194] Registered new tserver with Master: 1616c37cb2f54a8f9845332f263f9816 (127.0.97.1:35635)
I20260619 16:45:21.249413 11697 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:45639
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887385903558-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887385903558-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260619 16:45:21.270654 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu
/tmp/dist-test-taskEs2EH8/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887385903558-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887385903558-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887385903558-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887385903558-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:38923/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37823
--keytab_file=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887385903558-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42629
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887385903558-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887385903558-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887385903558-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 16:45:21 dist-test-slave-wvz7 krb5kdc[11221](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887520, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
W20260619 16:45:21.685494 11916 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:45:21.685839 11916 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:45:21.685981 11916 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:45:21.694751 11916 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:45:21.694868 11916 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:45:21.695022 11916 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260619 16:45:21.704743 11916 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42629
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887385903558-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887385903558-388-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.97.20:38923/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887385903558-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887385903558-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37823
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887385903558-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:45:21.706043 11916 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:45:21.707268 11916 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:45:21.722451 11931 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:45:21.722133 11933 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:45:21.723161 11916 server_base.cc:1034] running on GCE node
W20260619 16:45:21.723232 11930 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:45:21.723997 11916 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:45:21.725153 11916 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:45:21.726421 11916 hybrid_clock.cc:648] HybridClock initialized: now 1781887521726336 us; error 72 us; skew 500 ppm
I20260619 16:45:21.726788 11916 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:45:21 dist-test-slave-wvz7 krb5kdc[11221](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887521, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:45:21.730525 11916 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260619 16:45:21.732712 11916 webserver.cc:460] Webserver started at http://127.0.97.2:41085/ using document root <none> and password file <none>
I20260619 16:45:21.733385 11916 fs_manager.cc:359] Metadata directory not provided
I20260619 16:45:21.733492 11916 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:45:21.733842 11916 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 16:45:21 dist-test-slave-wvz7 krb5kdc[11221](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887521, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260619 16:45:21.751789 11916 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887385903558-388-0/minicluster-data/ts-1/data/instance:
uuid: "aa78df787c02441fa329d22afa3fb5b2"
format_stamp: "Formatted at 2026-06-19 16:45:21 on dist-test-slave-wvz7"
server_key: "231282baf8a2de7715b2578e4955f053"
server_key_iv: "13740689c9683c8503261bb5229b8ac5"
server_key_version: "kuduclusterkey@0"
I20260619 16:45:21.752995 11916 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887385903558-388-0/minicluster-data/ts-1/wal/instance:
uuid: "aa78df787c02441fa329d22afa3fb5b2"
format_stamp: "Formatted at 2026-06-19 16:45:21 on dist-test-slave-wvz7"
server_key: "231282baf8a2de7715b2578e4955f053"
server_key_iv: "13740689c9683c8503261bb5229b8ac5"
server_key_version: "kuduclusterkey@0"
I20260619 16:45:21.759939 11916 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260619 16:45:21.777643 11940 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:45:21.779347 11916 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260619 16:45:21.779507 11916 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887385903558-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887385903558-388-0/minicluster-data/ts-1/wal
uuid: "aa78df787c02441fa329d22afa3fb5b2"
format_stamp: "Formatted at 2026-06-19 16:45:21 on dist-test-slave-wvz7"
server_key: "231282baf8a2de7715b2578e4955f053"
server_key_iv: "13740689c9683c8503261bb5229b8ac5"
server_key_version: "kuduclusterkey@0"
I20260619 16:45:21.779714 11916 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887385903558-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887385903558-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887385903558-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:45:21.807898 11916 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:45:21.812060 11916 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:45:21.812325 11916 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:45:21.813253 11916 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:45:21.814585 11916 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:45:21.814659 11916 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:45:21.814764 11916 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:45:21.814815 11916 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:45:21.856643 11916 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:45687
I20260619 16:45:21.856698 12052 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:45687 every 8 connection(s)
I20260619 16:45:21.857924 11916 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887385903558-388-0/minicluster-data/ts-1/data/info.pb
I20260619 16:45:21.860482 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu as pid 11916
I20260619 16:45:21.860652 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887385903558-388-0/minicluster-data/ts-1/wal/instance
I20260619 16:45:21.861153 388 external_mini_cluster.cc:1428] Setting key 0938a890d288f45d3f987da4637fda79
Jun 19 16:45:21 dist-test-slave-wvz7 krb5kdc[11221](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887521, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
I20260619 16:45:21.895099 12056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:45:21.861169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37823 (local address 127.0.97.2:59169)
0619 16:45:21.873807 (+ 12638us) negotiation.cc:107] Waiting for socket to connect
0619 16:45:21.873871 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0619 16:45:21.876279 (+ 2408us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:45:21.877118 (+ 839us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:45:21.877130 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:45:21.877705 (+ 575us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:45:21.878576 (+ 871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:45:21.878602 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:45:21.879923 (+ 1321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:45:21.879928 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:45:21.881827 (+ 1899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:45:21.881841 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:45:21.882852 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:45:21.882859 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:45:21.883127 (+ 268us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:45:21.883902 (+ 775us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:45:21.883932 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:45:21.886780 (+ 2848us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:45:21.890648 (+ 3868us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:45:21.890658 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:45:21.890678 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:45:21.891243 (+ 565us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:45:21.891634 (+ 391us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:45:21.891638 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:45:21.891642 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:45:21.891877 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:45:21.892550 (+ 673us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:45:21.892558 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:45:21.892948 (+ 390us) client_negotiation.cc:769] Sending connection context
0619 16:45:21.893229 (+ 281us) client_negotiation.cc:240] Negotiation successful
0619 16:45:21.893768 (+ 539us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12280,"thread_start_us":12469,"threads_started":1}
I20260619 16:45:21.896387 12055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:45:21.860748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:59169 (local address 127.0.97.62:37823)
0619 16:45:21.873366 (+ 12618us) server_negotiation.cc:206] Beginning negotiation
0619 16:45:21.873372 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:45:21.874010 (+ 638us) server_negotiation.cc:407] Connection header received
0619 16:45:21.876607 (+ 2597us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:45:21.876613 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:45:21.876718 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:45:21.876833 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:45:21.878759 (+ 1926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:45:21.879761 (+ 1002us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:45:21.881980 (+ 2219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:45:21.882607 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:45:21.882743 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:45:21.887060 (+ 4317us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:45:21.887067 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:45:21.887072 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:45:21.887113 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:45:21.890476 (+ 3363us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:45:21.891388 (+ 912us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:45:21.891394 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:45:21.891398 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:45:21.891522 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:45:21.891997 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:45:21.892001 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:45:21.892004 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:45:21.892352 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:45:21.892479 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0619 16:45:21.895786 (+ 3307us) server_negotiation.cc:299] Negotiation successful
0619 16:45:21.896021 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12482,"thread_start_us":12350,"threads_started":1}
I20260619 16:45:21.897868 12053 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37823
I20260619 16:45:21.898233 12053 heartbeater.cc:464] Registering TS with master...
I20260619 16:45:21.900067 12053 heartbeater.cc:511] Master 127.0.97.62:37823 requested a full tablet report, sending...
I20260619 16:45:21.901669 11697 ts_manager.cc:194] Registered new tserver with Master: aa78df787c02441fa329d22afa3fb5b2 (127.0.97.2:45687)
I20260619 16:45:21.902747 11697 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:59169
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887385903558-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887385903558-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260619 16:45:21.916496 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu
/tmp/dist-test-taskEs2EH8/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887385903558-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887385903558-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887385903558-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887385903558-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:38923/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37823
--keytab_file=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887385903558-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42629
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887385903558-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887385903558-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887385903558-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:45:22.230579 12059 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:45:22.230835 12059 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:45:22.230945 12059 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:45:22.237906 12059 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:45:22.238016 12059 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:45:22.238174 12059 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260619 16:45:22.247915 12059 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42629
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887385903558-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887385903558-388-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.97.20:38923/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887385903558-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887385903558-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37823
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887385903558-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:45:22.249053 12059 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:45:22.250361 12059 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 16:45:22.252866 11910 heartbeater.cc:503] Master 127.0.97.62:37823 was elected leader, sending a full tablet report...
W20260619 16:45:22.263305 12064 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:45:22.263716 12067 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:45:22.263792 12065 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:45:22.265486 12059 server_base.cc:1034] running on GCE node
I20260619 16:45:22.266124 12059 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:45:22.267021 12059 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:45:22.268221 12059 hybrid_clock.cc:648] HybridClock initialized: now 1781887522268185 us; error 56 us; skew 500 ppm
I20260619 16:45:22.268523 12059 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:45:22 dist-test-slave-wvz7 krb5kdc[11221](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887522, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:45:22.271486 12059 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260619 16:45:22.272858 12059 webserver.cc:460] Webserver started at http://127.0.97.3:34905/ using document root <none> and password file <none>
I20260619 16:45:22.273512 12059 fs_manager.cc:359] Metadata directory not provided
I20260619 16:45:22.273612 12059 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:45:22.273947 12059 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 16:45:22 dist-test-slave-wvz7 krb5kdc[11221](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887522, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260619 16:45:22.289486 12059 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887385903558-388-0/minicluster-data/ts-2/data/instance:
uuid: "80c03fe837be487689d85d640233c133"
format_stamp: "Formatted at 2026-06-19 16:45:22 on dist-test-slave-wvz7"
server_key: "06da74f15655ca4571ea6f9fc8109934"
server_key_iv: "3ab80e7158db00826b4cfc69d6854e9f"
server_key_version: "kuduclusterkey@0"
I20260619 16:45:22.290365 12059 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887385903558-388-0/minicluster-data/ts-2/wal/instance:
uuid: "80c03fe837be487689d85d640233c133"
format_stamp: "Formatted at 2026-06-19 16:45:22 on dist-test-slave-wvz7"
server_key: "06da74f15655ca4571ea6f9fc8109934"
server_key_iv: "3ab80e7158db00826b4cfc69d6854e9f"
server_key_version: "kuduclusterkey@0"
I20260619 16:45:22.295758 12059 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260619 16:45:22.312566 12074 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:45:22.314188 12059 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260619 16:45:22.314356 12059 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887385903558-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887385903558-388-0/minicluster-data/ts-2/wal
uuid: "80c03fe837be487689d85d640233c133"
format_stamp: "Formatted at 2026-06-19 16:45:22 on dist-test-slave-wvz7"
server_key: "06da74f15655ca4571ea6f9fc8109934"
server_key_iv: "3ab80e7158db00826b4cfc69d6854e9f"
server_key_version: "kuduclusterkey@0"
I20260619 16:45:22.314555 12059 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887385903558-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887385903558-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887385903558-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:45:22.335460 12059 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:45:22.339282 12059 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:45:22.339552 12059 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:45:22.340458 12059 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:45:22.342237 12059 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:45:22.342311 12059 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:45:22.342437 12059 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:45:22.342484 12059 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:45:22.373366 12059 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:33431
I20260619 16:45:22.373420 12186 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:33431 every 8 connection(s)
I20260619 16:45:22.374619 12059 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887385903558-388-0/minicluster-data/ts-2/data/info.pb
I20260619 16:45:22.376210 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu as pid 12059
I20260619 16:45:22.376415 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887385903558-388-0/minicluster-data/ts-2/wal/instance
I20260619 16:45:22.376842 388 external_mini_cluster.cc:1428] Setting key 2cf05edb7c7fe06f5bc045b5e23ab31e
Jun 19 16:45:22 dist-test-slave-wvz7 krb5kdc[11221](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887522, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 16:45:22.394539 12055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:45:22.377299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:50239 (local address 127.0.97.62:37823)
0619 16:45:22.377490 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0619 16:45:22.377497 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:45:22.378617 (+ 1120us) server_negotiation.cc:407] Connection header received
0619 16:45:22.380514 (+ 1897us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:45:22.380518 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:45:22.380602 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:45:22.380704 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:45:22.382145 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:45:22.382876 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:45:22.384982 (+ 2106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:45:22.385414 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:45:22.385518 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:45:22.388807 (+ 3289us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:45:22.388813 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:45:22.388816 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:45:22.388850 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:45:22.391707 (+ 2857us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:45:22.392531 (+ 824us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:45:22.392536 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:45:22.392538 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:45:22.392655 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:45:22.393058 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:45:22.393063 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:45:22.393066 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:45:22.393370 (+ 304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:45:22.393481 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0619 16:45:22.394130 (+ 649us) server_negotiation.cc:299] Negotiation successful
0619 16:45:22.394306 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260619 16:45:22.395463 12189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:45:22.377697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37823 (local address 127.0.97.3:50239)
0619 16:45:22.378460 (+ 763us) negotiation.cc:107] Waiting for socket to connect
0619 16:45:22.378502 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0619 16:45:22.380253 (+ 1751us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:45:22.380934 (+ 681us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:45:22.380942 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:45:22.381342 (+ 400us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:45:22.381978 (+ 636us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:45:22.381997 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:45:22.383046 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:45:22.383050 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:45:22.384816 (+ 1766us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:45:22.384835 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:45:22.385630 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:45:22.385635 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:45:22.385895 (+ 260us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:45:22.386535 (+ 640us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:45:22.386557 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:45:22.388535 (+ 1978us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:45:22.391854 (+ 3319us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:45:22.391864 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:45:22.391882 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:45:22.392412 (+ 530us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:45:22.392734 (+ 322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:45:22.392737 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:45:22.392739 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:45:22.392926 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:45:22.393464 (+ 538us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:45:22.393469 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:45:22.393817 (+ 348us) client_negotiation.cc:769] Sending connection context
0619 16:45:22.394035 (+ 218us) client_negotiation.cc:240] Negotiation successful
0619 16:45:22.394413 (+ 378us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":498,"thread_start_us":315,"threads_started":1}
I20260619 16:45:22.396821 12187 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37823
I20260619 16:45:22.397099 12187 heartbeater.cc:464] Registering TS with master...
I20260619 16:45:22.397807 12187 heartbeater.cc:511] Master 127.0.97.62:37823 requested a full tablet report, sending...
I20260619 16:45:22.399099 11697 ts_manager.cc:194] Registered new tserver with Master: 80c03fe837be487689d85d640233c133 (127.0.97.3:33431)
I20260619 16:45:22.399821 11697 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:50239
I20260619 16:45:22.404438 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 16:45:22.420696 12055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:45:22.409243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55020 (local address 127.0.97.62:37823)
0619 16:45:22.409414 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0619 16:45:22.409420 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:45:22.409706 (+ 286us) server_negotiation.cc:407] Connection header received
0619 16:45:22.409874 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:45:22.409879 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:45:22.409947 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:45:22.410038 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:45:22.411366 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:45:22.412071 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:45:22.413265 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:45:22.413643 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:45:22.413727 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:45:22.415933 (+ 2206us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:45:22.415937 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:45:22.415941 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:45:22.415969 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:45:22.418351 (+ 2382us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:45:22.419136 (+ 785us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:45:22.419140 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:45:22.419143 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:45:22.419254 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:45:22.419597 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:45:22.419600 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:45:22.419602 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:45:22.419863 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:45:22.420045 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0619 16:45:22.420309 (+ 264us) server_negotiation.cc:299] Negotiation successful
0619 16:45:22.420462 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260619 16:45:22.424678 11697 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55020:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 16:45:22.432096 11697 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 16:45:22.468698 12202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:45:22.445134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45018 (local address 127.0.97.2:45687)
0619 16:45:22.459833 (+ 14699us) server_negotiation.cc:206] Beginning negotiation
0619 16:45:22.459843 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 16:45:22.461091 (+ 1248us) server_negotiation.cc:407] Connection header received
0619 16:45:22.461389 (+ 298us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:45:22.461397 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:45:22.461646 (+ 249us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:45:22.461851 (+ 205us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:45:22.462571 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:45:22.463733 (+ 1162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:45:22.465983 (+ 2250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:45:22.467371 (+ 1388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:45:22.467470 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:45:22.467646 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0619 16:45:22.468240 (+ 594us) server_negotiation.cc:299] Negotiation successful
0619 16:45:22.468398 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14513,"thread_start_us":14699,"threads_started":1}
I20260619 16:45:22.468753 12200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:45:22.444616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:45687 (local address 127.0.0.1:45018)
0619 16:45:22.460975 (+ 16359us) negotiation.cc:107] Waiting for socket to connect
0619 16:45:22.460993 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 16:45:22.461196 (+ 203us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:45:22.462041 (+ 845us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:45:22.462045 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:45:22.462070 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:45:22.462430 (+ 360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:45:22.462439 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:45:22.463868 (+ 1429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:45:22.463873 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:45:22.465841 (+ 1968us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:45:22.465856 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:45:22.467796 (+ 1940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:45:22.467802 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:45:22.468006 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:45:22.468058 (+ 52us) client_negotiation.cc:769] Sending connection context
0619 16:45:22.468182 (+ 124us) client_negotiation.cc:240] Negotiation successful
0619 16:45:22.468376 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16233,"mutex_wait_us":24,"thread_start_us":14321,"threads_started":1}
I20260619 16:45:22.469002 12201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:45:22.444811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41462 (local address 127.0.97.1:35635)
0619 16:45:22.460039 (+ 15228us) server_negotiation.cc:206] Beginning negotiation
0619 16:45:22.460046 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:45:22.460079 (+ 33us) server_negotiation.cc:407] Connection header received
0619 16:45:22.460195 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:45:22.460203 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:45:22.460582 (+ 379us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:45:22.460809 (+ 227us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:45:22.461987 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:45:22.463567 (+ 1580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:45:22.466200 (+ 2633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:45:22.467745 (+ 1545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:45:22.467867 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:45:22.468085 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0619 16:45:22.468510 (+ 425us) server_negotiation.cc:299] Negotiation successful
0619 16:45:22.468686 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15070,"thread_start_us":16180,"threads_started":1}
I20260619 16:45:22.469569 12199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:45:22.444485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:35635 (local address 127.0.0.1:41462)
0619 16:45:22.459293 (+ 14808us) negotiation.cc:107] Waiting for socket to connect
0619 16:45:22.459330 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0619 16:45:22.459497 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:45:22.461255 (+ 1758us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:45:22.461263 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:45:22.461297 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:45:22.461764 (+ 467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:45:22.461780 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:45:22.463707 (+ 1927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:45:22.463711 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:45:22.466038 (+ 2327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:45:22.466053 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:45:22.467949 (+ 1896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:45:22.467957 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:45:22.468137 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:45:22.468170 (+ 33us) client_negotiation.cc:769] Sending connection context
0619 16:45:22.468248 (+ 78us) client_negotiation.cc:240] Negotiation successful
0619 16:45:22.469229 (+ 981us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14564,"thread_start_us":16920,"threads_started":1}
I20260619 16:45:22.470124 12203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:45:22.446749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:33431 (local address 127.0.0.1:43760)
0619 16:45:22.462019 (+ 15270us) negotiation.cc:107] Waiting for socket to connect
0619 16:45:22.462034 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 16:45:22.462134 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:45:22.462688 (+ 554us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:45:22.462692 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:45:22.462715 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:45:22.463000 (+ 285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:45:22.463010 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:45:22.464745 (+ 1735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:45:22.464751 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:45:22.466785 (+ 2034us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:45:22.466796 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:45:22.469374 (+ 2578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:45:22.469380 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:45:22.469583 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:45:22.469616 (+ 33us) client_negotiation.cc:769] Sending connection context
0619 16:45:22.469698 (+ 82us) client_negotiation.cc:240] Negotiation successful
0619 16:45:22.469841 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15163,"mutex_wait_us":4,"thread_start_us":15021,"threads_started":1}
I20260619 16:45:22.470712 12204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:45:22.447331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43760 (local address 127.0.97.3:33431)
0619 16:45:22.456005 (+ 8674us) server_negotiation.cc:206] Beginning negotiation
0619 16:45:22.456016 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 16:45:22.462143 (+ 6127us) server_negotiation.cc:407] Connection header received
0619 16:45:22.462270 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:45:22.462276 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:45:22.462512 (+ 236us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:45:22.462699 (+ 187us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:45:22.463179 (+ 480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:45:22.464606 (+ 1427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:45:22.466967 (+ 2361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:45:22.468868 (+ 1901us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:45:22.468976 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:45:22.469186 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0619 16:45:22.470363 (+ 1177us) server_negotiation.cc:299] Negotiation successful
0619 16:45:22.470492 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7916,"thread_start_us":7741,"threads_started":1}
I20260619 16:45:22.472995 11988 tablet_service.cc:1467] Processing CreateTablet for tablet fc97229c78bb4319a3873a09d0aeba0f (DEFAULT_TABLE table=test-table [id=5b8988145db54823966422840c4e316f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:45:22.473336 12122 tablet_service.cc:1467] Processing CreateTablet for tablet fc97229c78bb4319a3873a09d0aeba0f (DEFAULT_TABLE table=test-table [id=5b8988145db54823966422840c4e316f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:45:22.473412 11845 tablet_service.cc:1467] Processing CreateTablet for tablet fc97229c78bb4319a3873a09d0aeba0f (DEFAULT_TABLE table=test-table [id=5b8988145db54823966422840c4e316f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:45:22.474532 11988 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fc97229c78bb4319a3873a09d0aeba0f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:45:22.474694 12122 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fc97229c78bb4319a3873a09d0aeba0f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:45:22.474658 11845 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fc97229c78bb4319a3873a09d0aeba0f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:45:22.485510 12205 tablet_bootstrap.cc:492] T fc97229c78bb4319a3873a09d0aeba0f P 80c03fe837be487689d85d640233c133: Bootstrap starting.
I20260619 16:45:22.488886 12207 tablet_bootstrap.cc:492] T fc97229c78bb4319a3873a09d0aeba0f P aa78df787c02441fa329d22afa3fb5b2: Bootstrap starting.
I20260619 16:45:22.489686 12205 tablet_bootstrap.cc:654] T fc97229c78bb4319a3873a09d0aeba0f P 80c03fe837be487689d85d640233c133: Neither blocks nor log segments found. Creating new log.
I20260619 16:45:22.489681 12206 tablet_bootstrap.cc:492] T fc97229c78bb4319a3873a09d0aeba0f P 1616c37cb2f54a8f9845332f263f9816: Bootstrap starting.
I20260619 16:45:22.491119 12205 log.cc:824] T fc97229c78bb4319a3873a09d0aeba0f P 80c03fe837be487689d85d640233c133: Log is configured to *not* fsync() on all Append() calls
I20260619 16:45:22.493050 12206 tablet_bootstrap.cc:654] T fc97229c78bb4319a3873a09d0aeba0f P 1616c37cb2f54a8f9845332f263f9816: Neither blocks nor log segments found. Creating new log.
I20260619 16:45:22.493274 12207 tablet_bootstrap.cc:654] T fc97229c78bb4319a3873a09d0aeba0f P aa78df787c02441fa329d22afa3fb5b2: Neither blocks nor log segments found. Creating new log.
I20260619 16:45:22.494330 12207 log.cc:824] T fc97229c78bb4319a3873a09d0aeba0f P aa78df787c02441fa329d22afa3fb5b2: Log is configured to *not* fsync() on all Append() calls
I20260619 16:45:22.494333 12206 log.cc:824] T fc97229c78bb4319a3873a09d0aeba0f P 1616c37cb2f54a8f9845332f263f9816: Log is configured to *not* fsync() on all Append() calls
I20260619 16:45:22.495536 12205 tablet_bootstrap.cc:492] T fc97229c78bb4319a3873a09d0aeba0f P 80c03fe837be487689d85d640233c133: No bootstrap required, opened a new log
I20260619 16:45:22.495801 12205 ts_tablet_manager.cc:1397] T fc97229c78bb4319a3873a09d0aeba0f P 80c03fe837be487689d85d640233c133: Time spent bootstrapping tablet: real 0.011s user 0.007s sys 0.000s
I20260619 16:45:22.497030 12207 tablet_bootstrap.cc:492] T fc97229c78bb4319a3873a09d0aeba0f P aa78df787c02441fa329d22afa3fb5b2: No bootstrap required, opened a new log
I20260619 16:45:22.497251 12207 ts_tablet_manager.cc:1397] T fc97229c78bb4319a3873a09d0aeba0f P aa78df787c02441fa329d22afa3fb5b2: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260619 16:45:22.497396 12206 tablet_bootstrap.cc:492] T fc97229c78bb4319a3873a09d0aeba0f P 1616c37cb2f54a8f9845332f263f9816: No bootstrap required, opened a new log
I20260619 16:45:22.497650 12206 ts_tablet_manager.cc:1397] T fc97229c78bb4319a3873a09d0aeba0f P 1616c37cb2f54a8f9845332f263f9816: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260619 16:45:22.504215 12207 raft_consensus.cc:348] T fc97229c78bb4319a3873a09d0aeba0f P aa78df787c02441fa329d22afa3fb5b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1616c37cb2f54a8f9845332f263f9816" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35635 } } peers { permanent_uuid: "aa78df787c02441fa329d22afa3fb5b2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45687 } } peers { permanent_uuid: "80c03fe837be487689d85d640233c133" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33431 } }
I20260619 16:45:22.504570 12207 raft_consensus.cc:374] T fc97229c78bb4319a3873a09d0aeba0f P aa78df787c02441fa329d22afa3fb5b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:45:22.504681 12207 raft_consensus.cc:729] T fc97229c78bb4319a3873a09d0aeba0f P aa78df787c02441fa329d22afa3fb5b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aa78df787c02441fa329d22afa3fb5b2, State: Initialized, Role: FOLLOWER
I20260619 16:45:22.505224 12207 consensus_queue.cc:260] T fc97229c78bb4319a3873a09d0aeba0f P aa78df787c02441fa329d22afa3fb5b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1616c37cb2f54a8f9845332f263f9816" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35635 } } peers { permanent_uuid: "aa78df787c02441fa329d22afa3fb5b2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45687 } } peers { permanent_uuid: "80c03fe837be487689d85d640233c133" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33431 } }
I20260619 16:45:22.506472 12053 heartbeater.cc:503] Master 127.0.97.62:37823 was elected leader, sending a full tablet report...
I20260619 16:45:22.507113 12207 ts_tablet_manager.cc:1428] T fc97229c78bb4319a3873a09d0aeba0f P aa78df787c02441fa329d22afa3fb5b2: Time spent starting tablet: real 0.010s user 0.004s sys 0.006s
I20260619 16:45:22.506057 12205 raft_consensus.cc:348] T fc97229c78bb4319a3873a09d0aeba0f P 80c03fe837be487689d85d640233c133 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1616c37cb2f54a8f9845332f263f9816" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35635 } } peers { permanent_uuid: "aa78df787c02441fa329d22afa3fb5b2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45687 } } peers { permanent_uuid: "80c03fe837be487689d85d640233c133" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33431 } }
I20260619 16:45:22.508204 12205 raft_consensus.cc:374] T fc97229c78bb4319a3873a09d0aeba0f P 80c03fe837be487689d85d640233c133 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:45:22.508317 12205 raft_consensus.cc:729] T fc97229c78bb4319a3873a09d0aeba0f P 80c03fe837be487689d85d640233c133 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80c03fe837be487689d85d640233c133, State: Initialized, Role: FOLLOWER
I20260619 16:45:22.508423 12206 raft_consensus.cc:348] T fc97229c78bb4319a3873a09d0aeba0f P 1616c37cb2f54a8f9845332f263f9816 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1616c37cb2f54a8f9845332f263f9816" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35635 } } peers { permanent_uuid: "aa78df787c02441fa329d22afa3fb5b2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45687 } } peers { permanent_uuid: "80c03fe837be487689d85d640233c133" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33431 } }
I20260619 16:45:22.508834 12206 raft_consensus.cc:374] T fc97229c78bb4319a3873a09d0aeba0f P 1616c37cb2f54a8f9845332f263f9816 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:45:22.508932 12206 raft_consensus.cc:729] T fc97229c78bb4319a3873a09d0aeba0f P 1616c37cb2f54a8f9845332f263f9816 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1616c37cb2f54a8f9845332f263f9816, State: Initialized, Role: FOLLOWER
I20260619 16:45:22.508971 12205 consensus_queue.cc:260] T fc97229c78bb4319a3873a09d0aeba0f P 80c03fe837be487689d85d640233c133 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1616c37cb2f54a8f9845332f263f9816" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35635 } } peers { permanent_uuid: "aa78df787c02441fa329d22afa3fb5b2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45687 } } peers { permanent_uuid: "80c03fe837be487689d85d640233c133" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33431 } }
I20260619 16:45:22.509476 12206 consensus_queue.cc:260] T fc97229c78bb4319a3873a09d0aeba0f P 1616c37cb2f54a8f9845332f263f9816 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1616c37cb2f54a8f9845332f263f9816" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35635 } } peers { permanent_uuid: "aa78df787c02441fa329d22afa3fb5b2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45687 } } peers { permanent_uuid: "80c03fe837be487689d85d640233c133" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33431 } }
I20260619 16:45:22.510095 12187 heartbeater.cc:503] Master 127.0.97.62:37823 was elected leader, sending a full tablet report...
I20260619 16:45:22.510775 12205 ts_tablet_manager.cc:1428] T fc97229c78bb4319a3873a09d0aeba0f P 80c03fe837be487689d85d640233c133: Time spent starting tablet: real 0.015s user 0.014s sys 0.000s
I20260619 16:45:22.512073 12206 ts_tablet_manager.cc:1428] T fc97229c78bb4319a3873a09d0aeba0f P 1616c37cb2f54a8f9845332f263f9816: Time spent starting tablet: real 0.014s user 0.011s sys 0.004s
W20260619 16:45:22.611035 12054 tablet.cc:2369] T fc97229c78bb4319a3873a09d0aeba0f P aa78df787c02441fa329d22afa3fb5b2: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 16:45:22.626088 12188 tablet.cc:2369] T fc97229c78bb4319a3873a09d0aeba0f P 80c03fe837be487689d85d640233c133: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 16:45:22.674660 12212 raft_consensus.cc:482] T fc97229c78bb4319a3873a09d0aeba0f P 80c03fe837be487689d85d640233c133 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 16:45:22.674937 12212 raft_consensus.cc:504] T fc97229c78bb4319a3873a09d0aeba0f P 80c03fe837be487689d85d640233c133 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1616c37cb2f54a8f9845332f263f9816" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35635 } } peers { permanent_uuid: "aa78df787c02441fa329d22afa3fb5b2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45687 } } peers { permanent_uuid: "80c03fe837be487689d85d640233c133" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33431 } }
I20260619 16:45:22.676501 12212 leader_election.cc:288] T fc97229c78bb4319a3873a09d0aeba0f P 80c03fe837be487689d85d640233c133 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1616c37cb2f54a8f9845332f263f9816 (127.0.97.1:35635), aa78df787c02441fa329d22afa3fb5b2 (127.0.97.2:45687)
I20260619 16:45:22.682816 12202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:45:22.676980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:43027 (local address 127.0.97.2:45687)
0619 16:45:22.677187 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0619 16:45:22.677192 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:45:22.677414 (+ 222us) server_negotiation.cc:407] Connection header received
0619 16:45:22.677908 (+ 494us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:45:22.677915 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:45:22.677994 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:45:22.678084 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:45:22.678452 (+ 368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:45:22.679369 (+ 917us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:45:22.681226 (+ 1857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:45:22.682018 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:45:22.682133 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:45:22.682222 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0619 16:45:22.682452 (+ 230us) server_negotiation.cc:299] Negotiation successful
0619 16:45:22.682569 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260619 16:45:22.682816 12189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:45:22.676937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:45687 (local address 127.0.97.3:43027)
0619 16:45:22.677308 (+ 371us) negotiation.cc:107] Waiting for socket to connect
0619 16:45:22.677332 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0619 16:45:22.677693 (+ 361us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:45:22.678115 (+ 422us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:45:22.678120 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:45:22.678151 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:45:22.678342 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:45:22.678352 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:45:22.679489 (+ 1137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:45:22.679492 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:45:22.681092 (+ 1600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:45:22.681104 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:45:22.682154 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:45:22.682160 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:45:22.682316 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:45:22.682341 (+ 25us) client_negotiation.cc:769] Sending connection context
0619 16:45:22.682419 (+ 78us) client_negotiation.cc:240] Negotiation successful
0619 16:45:22.682545 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":248}
I20260619 16:45:22.683375 12201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:45:22.676990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:52583 (local address 127.0.97.1:35635)
0619 16:45:22.677681 (+ 691us) server_negotiation.cc:206] Beginning negotiation
0619 16:45:22.677687 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:45:22.678313 (+ 626us) server_negotiation.cc:407] Connection header received
0619 16:45:22.678465 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:45:22.678470 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:45:22.678543 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:45:22.678632 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:45:22.679026 (+ 394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:45:22.679963 (+ 937us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:45:22.681764 (+ 1801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:45:22.682532 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:45:22.682651 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:45:22.682765 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0619 16:45:22.683071 (+ 306us) server_negotiation.cc:299] Negotiation successful
0619 16:45:22.683197 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260619 16:45:22.683900 12214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:45:22.676937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:35635 (local address 127.0.97.3:52583)
0619 16:45:22.678208 (+ 1271us) negotiation.cc:107] Waiting for socket to connect
0619 16:45:22.678230 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0619 16:45:22.678353 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:45:22.678640 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:45:22.678643 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:45:22.678660 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:45:22.678899 (+ 239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:45:22.678906 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:45:22.680109 (+ 1203us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:45:22.680114 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:45:22.681605 (+ 1491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:45:22.681617 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:45:22.682728 (+ 1111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:45:22.682733 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:45:22.682897 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:45:22.682927 (+ 30us) client_negotiation.cc:769] Sending connection context
0619 16:45:22.683545 (+ 618us) client_negotiation.cc:240] Negotiation successful
0619 16:45:22.683646 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1156,"mutex_wait_us":47,"thread_start_us":939,"threads_started":1}
I20260619 16:45:22.683743 12008 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc97229c78bb4319a3873a09d0aeba0f" candidate_uuid: "80c03fe837be487689d85d640233c133" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aa78df787c02441fa329d22afa3fb5b2" is_pre_election: true
I20260619 16:45:22.684248 12008 raft_consensus.cc:2454] T fc97229c78bb4319a3873a09d0aeba0f P aa78df787c02441fa329d22afa3fb5b2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 80c03fe837be487689d85d640233c133 in term 0.
I20260619 16:45:22.684891 12077 leader_election.cc:302] T fc97229c78bb4319a3873a09d0aeba0f P 80c03fe837be487689d85d640233c133 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 80c03fe837be487689d85d640233c133, aa78df787c02441fa329d22afa3fb5b2; no voters:
I20260619 16:45:22.684823 11865 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc97229c78bb4319a3873a09d0aeba0f" candidate_uuid: "80c03fe837be487689d85d640233c133" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1616c37cb2f54a8f9845332f263f9816" is_pre_election: true
I20260619 16:45:22.685228 12212 raft_consensus.cc:2781] T fc97229c78bb4319a3873a09d0aeba0f P 80c03fe837be487689d85d640233c133 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 16:45:22.685292 11865 raft_consensus.cc:2454] T fc97229c78bb4319a3873a09d0aeba0f P 1616c37cb2f54a8f9845332f263f9816 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 80c03fe837be487689d85d640233c133 in term 0.
I20260619 16:45:22.685389 12212 raft_consensus.cc:482] T fc97229c78bb4319a3873a09d0aeba0f P 80c03fe837be487689d85d640233c133 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 16:45:22.685479 12212 raft_consensus.cc:3037] T fc97229c78bb4319a3873a09d0aeba0f P 80c03fe837be487689d85d640233c133 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:45:22.688047 12212 raft_consensus.cc:504] T fc97229c78bb4319a3873a09d0aeba0f P 80c03fe837be487689d85d640233c133 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1616c37cb2f54a8f9845332f263f9816" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35635 } } peers { permanent_uuid: "aa78df787c02441fa329d22afa3fb5b2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45687 } } peers { permanent_uuid: "80c03fe837be487689d85d640233c133" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33431 } }
I20260619 16:45:22.688560 12212 leader_election.cc:288] T fc97229c78bb4319a3873a09d0aeba0f P 80c03fe837be487689d85d640233c133 [CANDIDATE]: Term 1 election: Requested vote from peers 1616c37cb2f54a8f9845332f263f9816 (127.0.97.1:35635), aa78df787c02441fa329d22afa3fb5b2 (127.0.97.2:45687)
I20260619 16:45:22.689040 12008 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc97229c78bb4319a3873a09d0aeba0f" candidate_uuid: "80c03fe837be487689d85d640233c133" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aa78df787c02441fa329d22afa3fb5b2"
I20260619 16:45:22.689042 11865 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc97229c78bb4319a3873a09d0aeba0f" candidate_uuid: "80c03fe837be487689d85d640233c133" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1616c37cb2f54a8f9845332f263f9816"
I20260619 16:45:22.689234 11865 raft_consensus.cc:3037] T fc97229c78bb4319a3873a09d0aeba0f P 1616c37cb2f54a8f9845332f263f9816 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:45:22.689253 12008 raft_consensus.cc:3037] T fc97229c78bb4319a3873a09d0aeba0f P aa78df787c02441fa329d22afa3fb5b2 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:45:22.692283 11865 raft_consensus.cc:2454] T fc97229c78bb4319a3873a09d0aeba0f P 1616c37cb2f54a8f9845332f263f9816 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 80c03fe837be487689d85d640233c133 in term 1.
I20260619 16:45:22.692298 12008 raft_consensus.cc:2454] T fc97229c78bb4319a3873a09d0aeba0f P aa78df787c02441fa329d22afa3fb5b2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 80c03fe837be487689d85d640233c133 in term 1.
I20260619 16:45:22.692726 12078 leader_election.cc:302] T fc97229c78bb4319a3873a09d0aeba0f P 80c03fe837be487689d85d640233c133 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 1616c37cb2f54a8f9845332f263f9816, 80c03fe837be487689d85d640233c133, aa78df787c02441fa329d22afa3fb5b2; no voters:
I20260619 16:45:22.693086 12212 raft_consensus.cc:2781] T fc97229c78bb4319a3873a09d0aeba0f P 80c03fe837be487689d85d640233c133 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:45:22.693601 12212 raft_consensus.cc:686] T fc97229c78bb4319a3873a09d0aeba0f P 80c03fe837be487689d85d640233c133 [term 1 LEADER]: Becoming Leader. State: Replica: 80c03fe837be487689d85d640233c133, State: Running, Role: LEADER
I20260619 16:45:22.694129 12212 consensus_queue.cc:237] T fc97229c78bb4319a3873a09d0aeba0f P 80c03fe837be487689d85d640233c133 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1616c37cb2f54a8f9845332f263f9816" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35635 } } peers { permanent_uuid: "aa78df787c02441fa329d22afa3fb5b2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45687 } } peers { permanent_uuid: "80c03fe837be487689d85d640233c133" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33431 } }
I20260619 16:45:22.699232 11697 catalog_manager.cc:5495] T fc97229c78bb4319a3873a09d0aeba0f P 80c03fe837be487689d85d640233c133 reported cstate change: term changed from 0 to 1, leader changed from <none> to 80c03fe837be487689d85d640233c133 (127.0.97.3). New cstate: current_term: 1 leader_uuid: "80c03fe837be487689d85d640233c133" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1616c37cb2f54a8f9845332f263f9816" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35635 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aa78df787c02441fa329d22afa3fb5b2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45687 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "80c03fe837be487689d85d640233c133" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33431 } health_report { overall_health: HEALTHY } } }
W20260619 16:45:22.714823 11911 tablet.cc:2369] T fc97229c78bb4319a3873a09d0aeba0f P 1616c37cb2f54a8f9845332f263f9816: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 16:45:22.749753 12204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:45:22.744283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43768 (local address 127.0.97.3:33431)
0619 16:45:22.744494 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0619 16:45:22.744500 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:45:22.744516 (+ 16us) server_negotiation.cc:407] Connection header received
0619 16:45:22.744596 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:45:22.744600 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:45:22.744672 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:45:22.744804 (+ 132us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 16:45:22.745175 (+ 371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:45:22.745923 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:45:22.747752 (+ 1829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:45:22.748130 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:45:22.748243 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:45:22.748702 (+ 459us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 16:45:22.749208 (+ 506us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 16:45:22.749331 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0619 16:45:22.749421 (+ 90us) server_negotiation.cc:299] Negotiation successful
0619 16:45:22.749493 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260619 16:45:22.757403 11865 raft_consensus.cc:1261] T fc97229c78bb4319a3873a09d0aeba0f P 1616c37cb2f54a8f9845332f263f9816 [term 1 FOLLOWER]: Refusing update from remote peer 80c03fe837be487689d85d640233c133: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:45:22.757402 12008 raft_consensus.cc:1261] T fc97229c78bb4319a3873a09d0aeba0f P aa78df787c02441fa329d22afa3fb5b2 [term 1 FOLLOWER]: Refusing update from remote peer 80c03fe837be487689d85d640233c133: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:45:22.758498 12215 consensus_queue.cc:1035] T fc97229c78bb4319a3873a09d0aeba0f P 80c03fe837be487689d85d640233c133 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1616c37cb2f54a8f9845332f263f9816" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:45:22.758827 12212 consensus_queue.cc:1035] T fc97229c78bb4319a3873a09d0aeba0f P 80c03fe837be487689d85d640233c133 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aa78df787c02441fa329d22afa3fb5b2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:45:22.770088 12219 mvcc.cc:205] Tried to move back new op lower bound from 7298611293204684800 to 7298611292959481856. Current Snapshot: MvccSnapshot[applied={T|T < 7298611293204684800}]
I20260619 16:45:22.781543 11696 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55020:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:45:22.781895 11696 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55020:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:45:22.785346 11696 catalog_manager.cc:5782] T 00000000000000000000000000000000 P cec89906d5fc4e5fa81d027ada4f6753: Sending DeleteTablet for 3 replicas of tablet fc97229c78bb4319a3873a09d0aeba0f
I20260619 16:45:22.786360 11845 tablet_service.cc:1514] Processing DeleteTablet for tablet fc97229c78bb4319a3873a09d0aeba0f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:45:22 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:41462
I20260619 16:45:22.786356 12122 tablet_service.cc:1514] Processing DeleteTablet for tablet fc97229c78bb4319a3873a09d0aeba0f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:45:22 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:43760
I20260619 16:45:22.786402 11988 tablet_service.cc:1514] Processing DeleteTablet for tablet fc97229c78bb4319a3873a09d0aeba0f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:45:22 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:45018
I20260619 16:45:22.789019 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu pid 11777
I20260619 16:45:22.797509 12229 tablet_replica.cc:330] T fc97229c78bb4319a3873a09d0aeba0f P aa78df787c02441fa329d22afa3fb5b2: stopping tablet replica
I20260619 16:45:22.797981 12229 raft_consensus.cc:2229] T fc97229c78bb4319a3873a09d0aeba0f P aa78df787c02441fa329d22afa3fb5b2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 16:45:22.798522 12229 raft_consensus.cc:2258] T fc97229c78bb4319a3873a09d0aeba0f P aa78df787c02441fa329d22afa3fb5b2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:45:22.799366 12228 tablet_replica.cc:330] T fc97229c78bb4319a3873a09d0aeba0f P 80c03fe837be487689d85d640233c133: stopping tablet replica
I20260619 16:45:22.799736 12228 raft_consensus.cc:2229] T fc97229c78bb4319a3873a09d0aeba0f P 80c03fe837be487689d85d640233c133 [term 1 LEADER]: Raft consensus shutting down.
I20260619 16:45:22.800429 12228 raft_consensus.cc:2258] T fc97229c78bb4319a3873a09d0aeba0f P 80c03fe837be487689d85d640233c133 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:45:22.801160 12227 tablet_replica.cc:330] T fc97229c78bb4319a3873a09d0aeba0f P 1616c37cb2f54a8f9845332f263f9816: stopping tablet replica
I20260619 16:45:22.801553 12227 raft_consensus.cc:2229] T fc97229c78bb4319a3873a09d0aeba0f P 1616c37cb2f54a8f9845332f263f9816 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 16:45:22.802071 12227 raft_consensus.cc:2258] T fc97229c78bb4319a3873a09d0aeba0f P 1616c37cb2f54a8f9845332f263f9816 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:45:22.802325 12228 ts_tablet_manager.cc:1905] T fc97229c78bb4319a3873a09d0aeba0f P 80c03fe837be487689d85d640233c133: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:45:22.804206 12229 ts_tablet_manager.cc:1905] T fc97229c78bb4319a3873a09d0aeba0f P aa78df787c02441fa329d22afa3fb5b2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:45:22.807276 12228 ts_tablet_manager.cc:1918] T fc97229c78bb4319a3873a09d0aeba0f P 80c03fe837be487689d85d640233c133: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 16:45:22.807428 12228 log.cc:1196] T fc97229c78bb4319a3873a09d0aeba0f P 80c03fe837be487689d85d640233c133: Deleting WAL directory at /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887385903558-388-0/minicluster-data/ts-2/wal/wals/fc97229c78bb4319a3873a09d0aeba0f
I20260619 16:45:22.807956 12228 ts_tablet_manager.cc:1939] T fc97229c78bb4319a3873a09d0aeba0f P 80c03fe837be487689d85d640233c133: Deleting consensus metadata
I20260619 16:45:22.808736 12227 ts_tablet_manager.cc:1905] T fc97229c78bb4319a3873a09d0aeba0f P 1616c37cb2f54a8f9845332f263f9816: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:45:22.809666 11681 catalog_manager.cc:4841] TS 80c03fe837be487689d85d640233c133 (127.0.97.3:33431): tablet fc97229c78bb4319a3873a09d0aeba0f (table test-table [id=5b8988145db54823966422840c4e316f]) successfully deleted
I20260619 16:45:22.810787 12229 ts_tablet_manager.cc:1918] T fc97229c78bb4319a3873a09d0aeba0f P aa78df787c02441fa329d22afa3fb5b2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 16:45:22.810966 12229 log.cc:1196] T fc97229c78bb4319a3873a09d0aeba0f P aa78df787c02441fa329d22afa3fb5b2: Deleting WAL directory at /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887385903558-388-0/minicluster-data/ts-1/wal/wals/fc97229c78bb4319a3873a09d0aeba0f
I20260619 16:45:22.811445 12229 ts_tablet_manager.cc:1939] T fc97229c78bb4319a3873a09d0aeba0f P aa78df787c02441fa329d22afa3fb5b2: Deleting consensus metadata
I20260619 16:45:22.813643 11683 catalog_manager.cc:4841] TS aa78df787c02441fa329d22afa3fb5b2 (127.0.97.2:45687): tablet fc97229c78bb4319a3873a09d0aeba0f (table test-table [id=5b8988145db54823966422840c4e316f]) successfully deleted
I20260619 16:45:22.814252 12227 ts_tablet_manager.cc:1918] T fc97229c78bb4319a3873a09d0aeba0f P 1616c37cb2f54a8f9845332f263f9816: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 16:45:22.814388 12227 log.cc:1196] T fc97229c78bb4319a3873a09d0aeba0f P 1616c37cb2f54a8f9845332f263f9816: Deleting WAL directory at /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887385903558-388-0/minicluster-data/ts-0/wal/wals/fc97229c78bb4319a3873a09d0aeba0f
I20260619 16:45:22.814742 12227 ts_tablet_manager.cc:1939] T fc97229c78bb4319a3873a09d0aeba0f P 1616c37cb2f54a8f9845332f263f9816: Deleting consensus metadata
I20260619 16:45:22.816403 11684 catalog_manager.cc:4841] TS 1616c37cb2f54a8f9845332f263f9816 (127.0.97.1:35635): tablet fc97229c78bb4319a3873a09d0aeba0f (table test-table [id=5b8988145db54823966422840c4e316f]) successfully deleted
Jun 19 16:45:22 dist-test-slave-wvz7 krb5kdc[11221](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887518, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260619 16:45:22.823755 12201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:45:22.790568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41474 (local address 127.0.97.1:35635)
0619 16:45:22.790744 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0619 16:45:22.790750 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:45:22.808107 (+ 17357us) server_negotiation.cc:407] Connection header received
0619 16:45:22.808587 (+ 480us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:45:22.808593 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:45:22.808672 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:45:22.808749 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:45:22.810546 (+ 1797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:45:22.811647 (+ 1101us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:45:22.813585 (+ 1938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:45:22.814137 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:45:22.814212 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:45:22.818511 (+ 4299us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:45:22.818522 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:45:22.818535 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:45:22.818567 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:45:22.821227 (+ 2660us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:45:22.821887 (+ 660us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:45:22.821894 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:45:22.821897 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:45:22.822015 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:45:22.822370 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:45:22.822374 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:45:22.822376 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:45:22.822696 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:45:22.822967 (+ 271us) server_negotiation.cc:1035] Waiting for connection context
0619 16:45:22.823229 (+ 262us) server_negotiation.cc:299] Negotiation successful
0619 16:45:22.823452 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260619 16:45:22.824414 11905 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:45:22.913103 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu with pid 11777
I20260619 16:45:22.927472 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu pid 11916
Jun 19 16:45:22 dist-test-slave-wvz7 krb5kdc[11221](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887518, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260619 16:45:22.941645 12202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:45:22.928179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41654 (local address 127.0.97.2:45687)
0619 16:45:22.928336 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0619 16:45:22.928341 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:45:22.928419 (+ 78us) server_negotiation.cc:407] Connection header received
0619 16:45:22.928595 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:45:22.928598 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:45:22.928660 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:45:22.928742 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:45:22.929943 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:45:22.930650 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:45:22.931921 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:45:22.932306 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:45:22.932408 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:45:22.936345 (+ 3937us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:45:22.936358 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:45:22.936373 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:45:22.936417 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:45:22.939110 (+ 2693us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:45:22.939717 (+ 607us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:45:22.939723 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:45:22.939726 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:45:22.939884 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:45:22.940214 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:45:22.940219 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:45:22.940223 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:45:22.940574 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:45:22.940836 (+ 262us) server_negotiation.cc:1035] Waiting for connection context
0619 16:45:22.941123 (+ 287us) server_negotiation.cc:299] Negotiation successful
0619 16:45:22.941315 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260619 16:45:22.942513 12048 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:45:23.031914 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu with pid 11916
I20260619 16:45:23.044566 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu pid 12059
Jun 19 16:45:23 dist-test-slave-wvz7 krb5kdc[11221](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887518, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260619 16:45:23.059813 12204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:45:23.045395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44958 (local address 127.0.97.3:33431)
0619 16:45:23.045594 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0619 16:45:23.045600 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:45:23.045622 (+ 22us) server_negotiation.cc:407] Connection header received
0619 16:45:23.045771 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:45:23.045777 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:45:23.045840 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:45:23.045939 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:45:23.047290 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:45:23.048078 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:45:23.049399 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:45:23.049881 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:45:23.049999 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:45:23.054194 (+ 4195us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:45:23.054205 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:45:23.054222 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:45:23.054256 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:45:23.057115 (+ 2859us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:45:23.057760 (+ 645us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:45:23.057769 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:45:23.057772 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:45:23.057932 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:45:23.058262 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:45:23.058267 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:45:23.058271 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:45:23.058588 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:45:23.058864 (+ 276us) server_negotiation.cc:1035] Waiting for connection context
0619 16:45:23.059157 (+ 293us) server_negotiation.cc:299] Negotiation successful
0619 16:45:23.059378 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260619 16:45:23.060443 12182 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:45:23.176733 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu with pid 12059
I20260619 16:45:23.191082 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu pid 11665
I20260619 16:45:23.192170 11727 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:45:23.300946 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu with pid 11665
I20260619 16:45:23.319872 388 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260619 16:45:23.654826 388 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260619 16:45:23.655043 388 mini_ranger.cc:66] Stopping Ranger...
I20260619 16:45:23.999667 388 mini_ranger.cc:68] Stopped Ranger
2026-06-19 16:45:23.999 UTC [11251] LOG: received smart shutdown request
2026-06-19 16:45:24.002 UTC [11251] LOG: background worker "logical replication launcher" (PID 11260) exited with exit code 1
2026-06-19 16:45:24.002 UTC [11255] LOG: shutting down
2026-06-19 16:45:24.139 UTC [11251] LOG: database system is shut down
2026-06-19T16:45:24Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (47388 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887385903558-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 16:45:24 dist-test-slave-wvz7 krb5kdc[12243](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 16:45:24 dist-test-slave-wvz7 krb5kdc[12243](info): set up 2 sockets
Jun 19 16:45:24 dist-test-slave-wvz7 krb5kdc[12243](info): commencing operation
krb5kdc: starting...
W20260619 16:45:25.245756 388 mini_kdc.cc:121] Time spent starting KDC: real 0.972s user 0.007s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887385903558-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887385903558-388-0/krb5kdc/test-user.keytab.
Jun 19 16:45:25 dist-test-slave-wvz7 krb5kdc[12243](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887525, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T16:45:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T16:45:25Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887385903558-388-0/krb5kdc/rangeradmin_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887385903558-388-0/krb5kdc/rangeradmin_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887385903558-388-0/krb5kdc/rangerlookup_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887385903558-388-0/krb5kdc/rangerlookup_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887385903558-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887385903558-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
I20260619 16:45:25.523568 388 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887385903558-388-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskEs2EH8/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887385903558-388-0/minicluster-data/postgres -l logfile start
2026-06-19 16:45:27.715 UTC [12273] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-06-19 16:45:27.715 UTC [12273] LOG: listening on IPv4 address "127.0.97.20", port 48611
2026-06-19 16:45:27.718 UTC [12273] LOG: listening on Unix socket "/tmp/.s.PGSQL.48611"
2026-06-19 16:45:27.738 UTC [12276] LOG: database system was shut down at 2026-06-19 16:45:26 UTC
2026-06-19 16:45:27.741 UTC [12273] LOG: database system is ready to accept connections
I20260619 16:45:28.637501 388 mini_postgres.cc:96] Postgres bound to 48611
2026-06-19 16:45:28.653 UTC [12284] FATAL: database "slave" does not exist
127.0.97.20:48611 - accepting connections
I20260619 16:45:28.653539 388 mini_ranger.cc:161] Starting Ranger...
I20260619 16:45:28.676600 388 mini_ranger.cc:84] Created miniranger Postgres user
I20260619 16:45:29.090822 388 mini_ranger.cc:87] Created ranger Postgres database
I20260619 16:45:29.090914 388 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887385903558-388-0/minicluster-data/ranger-admin
2026-06-19 16:45:29,363 [I] DB FLAVOR :POSTGRES
2026-06-19 16:45:29,363 [I] --------- Verifying Ranger DB connection ---------
2026-06-19 16:45:29,363 [I] Checking connection..
2026-06-19 16:45:29,363 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskEs2EH8/build/asan/bin/postgresql.jar:/tmp/dist-test-taskEs2EH8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:48611/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-06-19 16:45:29,596 [I] Checking connection passed.
2026-06-19 16:45:29,596 [I] --------- Verifying version history table ---------
2026-06-19 16:45:29,597 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskEs2EH8/build/asan/bin/postgresql.jar:/tmp/dist-test-taskEs2EH8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:48611/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-19 16:45:29,823 [I] Table x_db_version_h does not exist in database ranger
2026-06-19 16:45:29,824 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskEs2EH8/build/asan/bin/postgresql.jar:/tmp/dist-test-taskEs2EH8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:48611/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-19 16:45:30,057 [I] Table x_db_version_h does not exist in database ranger
2026-06-19 16:45:30,058 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-06-19 16:45:30,058 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskEs2EH8/build/asan/bin/postgresql.jar:/tmp/dist-test-taskEs2EH8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:48611/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskEs2EH8/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-06-19 16:45:30.303 UTC [12348] WARNING: there is no transaction in progress
2026-06-19 16:45:30,309 [I] create_dbversion_catalog.sql file imported successfully
2026-06-19 16:45:30,309 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskEs2EH8/build/asan/bin/postgresql.jar:/tmp/dist-test-taskEs2EH8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:48611/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-19 16:45:30,561 [I] Table x_db_version_h already exists in database 'ranger'
2026-06-19 16:45:30,561 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-19 16:45:30,561 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskEs2EH8/build/asan/bin/postgresql.jar:/tmp/dist-test-taskEs2EH8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:48611/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-06-19 16:45:30,788 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskEs2EH8/build/asan/bin/postgresql.jar:/tmp/dist-test-taskEs2EH8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:48611/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-06-19 16:45:31,029 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskEs2EH8/build/asan/bin/postgresql.jar:/tmp/dist-test-taskEs2EH8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:48611/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-wvz7.c.gcp-upstream.internal','N') ;"
2026-06-19 16:45:31,263 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskEs2EH8/build/asan/bin/postgresql.jar:/tmp/dist-test-taskEs2EH8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:48611/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-19 16:45:31,516 [I] Table x_portal_user does not exist in database ranger
2026-06-19 16:45:31,516 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskEs2EH8/build/asan/bin/postgresql.jar:/tmp/dist-test-taskEs2EH8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:48611/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-19 16:45:31,764 [I] Table x_policy_ref_group does not exist in database ranger
2026-06-19 16:45:31,764 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskEs2EH8/build/asan/bin/postgresql.jar:/tmp/dist-test-taskEs2EH8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:48611/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-19 16:45:32,008 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskEs2EH8/build/asan/bin/postgresql.jar:/tmp/dist-test-taskEs2EH8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:48611/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-19 16:45:32,254 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-06-19 16:45:32,254 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskEs2EH8/build/asan/bin/postgresql.jar:/tmp/dist-test-taskEs2EH8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:48611/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskEs2EH8/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-06-19 16:45:33.151 UTC [12456] WARNING: there is no transaction in progress
2026-06-19 16:45:33.165 UTC [12456] WARNING: there is no transaction in progress
2026-06-19 16:45:33.180 UTC [12456] WARNING: there is no transaction in progress
2026-06-19 16:45:33.195 UTC [12456] WARNING: there is no transaction in progress
2026-06-19 16:45:33.210 UTC [12456] WARNING: there is no transaction in progress
2026-06-19 16:45:33.226 UTC [12456] WARNING: there is no transaction in progress
2026-06-19 16:45:33.275 UTC [12456] WARNING: there is no transaction in progress
2026-06-19 16:45:33.280 UTC [12456] WARNING: there is no transaction in progress
2026-06-19 16:45:33.292 UTC [12456] WARNING: there is no transaction in progress
2026-06-19 16:45:33.301 UTC [12456] WARNING: there is no transaction in progress
2026-06-19 16:45:33.310 UTC [12456] WARNING: there is no transaction in progress
2026-06-19 16:45:33.322 UTC [12456] WARNING: there is no transaction in progress
2026-06-19 16:45:33.330 UTC [12456] WARNING: there is no transaction in progress
2026-06-19 16:45:33.336 UTC [12456] WARNING: there is no transaction in progress
2026-06-19 16:45:33.342 UTC [12456] WARNING: there is no transaction in progress
2026-06-19 16:45:33.347 UTC [12456] WARNING: there is no transaction in progress
2026-06-19 16:45:33,900 [I] ranger_core_db_postgres.sql file imported successfully
2026-06-19 16:45:33,900 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskEs2EH8/build/asan/bin/postgresql.jar:/tmp/dist-test-taskEs2EH8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:48611/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-06-19 16:45:34,131 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-06-19 16:45:34,131 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskEs2EH8/build/asan/bin/postgresql.jar:/tmp/dist-test-taskEs2EH8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:48611/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-19 16:45:34,355 [I] Table x_portal_user already exists in database 'ranger'
2026-06-19 16:45:34,355 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskEs2EH8/build/asan/bin/postgresql.jar:/tmp/dist-test-taskEs2EH8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:48611/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-19 16:45:34,597 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-06-19 16:45:34,598 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskEs2EH8/build/asan/bin/postgresql.jar:/tmp/dist-test-taskEs2EH8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:48611/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-19 16:45:34,841 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskEs2EH8/build/asan/bin/postgresql.jar:/tmp/dist-test-taskEs2EH8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:48611/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-19 16:45:35,091 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskEs2EH8/build/asan/bin/postgresql.jar:/tmp/dist-test-taskEs2EH8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:48611/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-wvz7.c.gcp-upstream.internal';"
2026-06-19 16:45:35,333 [I] CORE_DB_SCHEMA import status has been updated
2026-06-19 16:45:35,333 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskEs2EH8/build/asan/bin/postgresql.jar:/tmp/dist-test-taskEs2EH8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:48611/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-06-19 16:45:35,558 [I] DB_PATCHES have already been applied
I20260619 16:45:35.564352 388 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887385903558-388-0/minicluster-data/ranger-admin:/tmp/dist-test-taskEs2EH8/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskEs2EH8/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskEs2EH8/build/asan/bin/postgresql.jar:/tmp/dist-test-taskEs2EH8/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260619 16:45:35.564435 388 mini_ranger.cc:193] Using host: 127.0.97.20
I20260619 16:45:35.577318 388 mini_ranger.cc:227] Ranger admin URL: http://127.0.97.20:49709
Jun 19, 2026 4:45:35 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 19, 2026 4:45:35 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 19, 2026 4:45:35 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887385903558-388-0/minicluster-data/ranger-admin/ews/webapp
Jun 19, 2026 4:45:35 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887385903558-388-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Jun 19, 2026 4:45:35 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887385903558-388-0/minicluster-data/ranger-admin/ews/webapp] .....
Jun 19, 2026 4:45:36 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887385903558-388-0/minicluster-data/ranger-admin/ews/webapp].
Jun 19, 2026 4:45:36 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260619 16:45:36.588117 388 mini_ranger.cc:160] Time spent starting Ranger: real 7.935s user 0.005s sys 0.030s
Jun 19, 2026 4:45:54 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Jun 19, 2026 4:45:54 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Jun 19, 2026 4:45:54 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Jun 19, 2026 4:45:54 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Jun 19, 2026 4:45:54 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Jun 19, 2026 4:45:54 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Jun 19, 2026 4:45:54 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Jun 19, 2026 4:45:54 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Jun 19, 2026 4:45:54 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Jun 19, 2026 4:45:54 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Jun 19, 2026 4:45:54 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Jun 19, 2026 4:45:54 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Jun 19, 2026 4:45:54 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Jun 19, 2026 4:45:54 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Jun 19, 2026 4:45:54 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Jun 19, 2026 4:45:54 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Jun 19, 2026 4:45:54 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Jun 19, 2026 4:45:55 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260619 16:45:56.091655 388 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887385903558-388-0/krb5kdc/rangerkms_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887385903558-388-0/krb5kdc/rangerkms_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.97.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.97.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887385903558-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887385903558-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Jun 19 16:45:56 dist-test-slave-wvz7 krb5kdc[12243](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887556, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260619 16:45:56.248049 388 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260619 16:45:56.266521 388 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260619 16:45:56.963959 388 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260619 16:45:56.964080 388 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887385903558-388-0/minicluster-data/ranger-kms
I20260619 16:45:56.964126 388 mini_ranger_kms.cc:227] Using postgres at 127.0.97.20:48611
2026-06-19 16:45:57,290 [I] DB FLAVOR :POSTGRES
2026-06-19 16:45:57,290 [I] --------- Verifying Ranger DB connection ---------
2026-06-19 16:45:57,290 [I] Checking connection
2026-06-19 16:45:57,290 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskEs2EH8/build/asan/bin/postgresql.jar:/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887385903558-388-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:48611/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-06-19 16:45:57,564 [I] connection success
2026-06-19 16:45:57,564 [I] --------- Verifying Ranger DB tables ---------
2026-06-19 16:45:57,564 [I] Verifying table ranger_masterkey in database rangerkms
2026-06-19 16:45:57,565 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskEs2EH8/build/asan/bin/postgresql.jar:/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887385903558-388-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:48611/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-06-19 16:45:57,818 [I] Table ranger_masterkey does not exist in database rangerkms
2026-06-19 16:45:57,818 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-19 16:45:57,819 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-06-19 16:45:57,819 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskEs2EH8/build/asan/bin/postgresql.jar:/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887385903558-388-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:48611/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887385903558-388-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-06-19 16:45:58,068 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260619 16:45:58.502218 388 mini_ranger_kms.cc:310] Created kms service
I20260619 16:45:58.689368 388 mini_ranger_kms.cc:326] Created kudu user
I20260619 16:45:58.733592 388 mini_ranger_kms.cc:343] Created rangerkms user
Jun 19 16:45:58 dist-test-slave-wvz7 krb5kdc[12243](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887556, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260619 16:45:59.113009 388 mini_ranger_kms.cc:384] Added ranger policy
I20260619 16:45:59.113303 388 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887385903558-388-0/minicluster-data/ranger-kms:/tmp/dist-test-taskEs2EH8/build/asan/bin/postgresql.jar:/tmp/dist-test-taskEs2EH8/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskEs2EH8/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260619 16:45:59.113359 388 mini_ranger_kms.cc:240] Using host: 127.0.97.20
I20260619 16:45:59.130033 388 mini_ranger_kms.cc:276] Ranger KMS PID: 12636
I20260619 16:45:59.130146 388 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.97.20:60203
Jun 19, 2026 4:46:00 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 19, 2026 4:46:00 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 19, 2026 4:46:00 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887385903558-388-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Jun 19, 2026 4:46:00 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887385903558-388-0/minicluster-data/ranger-kms/ews/webapp] .....
Jun 19, 2026 4:46:00 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887385903558-388-0/minicluster-data/ranger-kms/ews/webapp].
Jun 19, 2026 4:46:00 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Jun 19 16:46:01 dist-test-slave-wvz7 krb5kdc[12243](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781887561, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.97.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:46:01.200562 388 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 4.953s user 0.068s sys 0.269s
I20260619 16:46:01.200762 388 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260619 16:46:01.200915 388 mini_ranger_kms.cc:401] 127.0.97.20:60203/kms/v1/keys
Jun 19 16:46:01 dist-test-slave-wvz7 krb5kdc[12243](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887561, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.97.20@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
Jun 19 16:46:01 dist-test-slave-wvz7 krb5kdc[12243](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887561, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.97.20@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
Jun 19 16:46:04 dist-test-slave-wvz7 krb5kdc[12243](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887564, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887385903558-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887385903558-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260619 16:46:04.237143 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu
/tmp/dist-test-taskEs2EH8/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887385903558-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887385903558-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887385903558-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887385903558-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:43371
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887385903558-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40517
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:43371
--ranger_config_path=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887385903558-388-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:60203/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887385903558-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887385903558-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887385903558-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:46:04.546988 12687 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:46:04.547260 12687 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:46:04.547369 12687 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:46:04.554376 12687 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:46:04.554487 12687 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:46:04.554550 12687 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:46:04.554620 12687 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:46:04.554679 12687 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:46:04.564435 12687 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40517
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887385903558-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887385903558-388-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.97.20:60203/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:43371
--ranger_config_path=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887385903558-388-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887385903558-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:43371
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887385903558-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887385903558-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:46:04.565603 12687 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:46:04.566869 12687 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:46:04.579241 12692 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:46:04.579267 12693 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:46:04.579309 12695 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:46:04.579789 12687 server_base.cc:1034] running on GCE node
I20260619 16:46:04.580727 12687 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:46:04.582386 12687 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:46:04.583614 12687 hybrid_clock.cc:648] HybridClock initialized: now 1781887564583572 us; error 29 us; skew 500 ppm
I20260619 16:46:04.583995 12687 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:46:04 dist-test-slave-wvz7 krb5kdc[12243](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887564, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:46:04.591903 12687 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260619 16:46:04.593672 12687 webserver.cc:460] Webserver started at http://127.0.97.62:34833/ using document root <none> and password file <none>
I20260619 16:46:04.594533 12687 fs_manager.cc:359] Metadata directory not provided
I20260619 16:46:04.594662 12687 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:46:04.595018 12687 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 16:46:04 dist-test-slave-wvz7 krb5kdc[12243](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887564, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260619 16:46:04.646075 12687 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887385903558-388-0/minicluster-data/master-0/data/instance:
uuid: "beda6ee725e9415687ecdf160264704b"
format_stamp: "Formatted at 2026-06-19 16:46:04 on dist-test-slave-wvz7"
server_key: "6c7e90a8d42aeafd13c5e11cc6e00071"
server_key_iv: "15ac713d0d631840c0ef1a940b474e91"
server_key_version: "kuduclusterkey@0"
I20260619 16:46:04.647109 12687 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887385903558-388-0/minicluster-data/master-0/wal/instance:
uuid: "beda6ee725e9415687ecdf160264704b"
format_stamp: "Formatted at 2026-06-19 16:46:04 on dist-test-slave-wvz7"
server_key: "6c7e90a8d42aeafd13c5e11cc6e00071"
server_key_iv: "15ac713d0d631840c0ef1a940b474e91"
server_key_version: "kuduclusterkey@0"
I20260619 16:46:04.654099 12687 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.002s sys 0.005s
I20260619 16:46:04.670912 12702 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:46:04.672482 12687 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260619 16:46:04.672653 12687 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887385903558-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887385903558-388-0/minicluster-data/master-0/wal
uuid: "beda6ee725e9415687ecdf160264704b"
format_stamp: "Formatted at 2026-06-19 16:46:04 on dist-test-slave-wvz7"
server_key: "6c7e90a8d42aeafd13c5e11cc6e00071"
server_key_iv: "15ac713d0d631840c0ef1a940b474e91"
server_key_version: "kuduclusterkey@0"
I20260619 16:46:04.672865 12687 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887385903558-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887385903558-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887385903558-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:46:04.685418 12687 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:46:04.689280 12687 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:46:04.689563 12687 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:46:04.707203 12687 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:43371
I20260619 16:46:04.707233 12753 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:43371 every 8 connection(s)
I20260619 16:46:04.708520 12687 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887385903558-388-0/minicluster-data/master-0/data/info.pb
I20260619 16:46:04.710695 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu as pid 12687
I20260619 16:46:04.710912 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887385903558-388-0/minicluster-data/master-0/wal/instance
I20260619 16:46:04.711690 388 external_mini_cluster.cc:1428] Setting key 4654ba82fe00c0d739efcb36ecca2a5b
I20260619 16:46:04.712412 12754 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:46:04.727825 12754 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P beda6ee725e9415687ecdf160264704b: Bootstrap starting.
I20260619 16:46:04.732024 12754 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P beda6ee725e9415687ecdf160264704b: Neither blocks nor log segments found. Creating new log.
I20260619 16:46:04.733436 12754 log.cc:824] T 00000000000000000000000000000000 P beda6ee725e9415687ecdf160264704b: Log is configured to *not* fsync() on all Append() calls
I20260619 16:46:04.737432 12754 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P beda6ee725e9415687ecdf160264704b: No bootstrap required, opened a new log
Jun 19 16:46:04 dist-test-slave-wvz7 krb5kdc[12243](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887564, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 16:46:04.747572 12754 raft_consensus.cc:348] T 00000000000000000000000000000000 P beda6ee725e9415687ecdf160264704b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "beda6ee725e9415687ecdf160264704b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43371 } }
I20260619 16:46:04.748050 12754 raft_consensus.cc:374] T 00000000000000000000000000000000 P beda6ee725e9415687ecdf160264704b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:46:04.748167 12754 raft_consensus.cc:729] T 00000000000000000000000000000000 P beda6ee725e9415687ecdf160264704b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: beda6ee725e9415687ecdf160264704b, State: Initialized, Role: FOLLOWER
I20260619 16:46:04.748936 12754 consensus_queue.cc:260] T 00000000000000000000000000000000 P beda6ee725e9415687ecdf160264704b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "beda6ee725e9415687ecdf160264704b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43371 } }
I20260619 16:46:04.749231 12754 raft_consensus.cc:388] T 00000000000000000000000000000000 P beda6ee725e9415687ecdf160264704b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:46:04.749382 12754 raft_consensus.cc:482] T 00000000000000000000000000000000 P beda6ee725e9415687ecdf160264704b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:46:04.749569 12754 raft_consensus.cc:3037] T 00000000000000000000000000000000 P beda6ee725e9415687ecdf160264704b [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:46:04.752596 12754 raft_consensus.cc:504] T 00000000000000000000000000000000 P beda6ee725e9415687ecdf160264704b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "beda6ee725e9415687ecdf160264704b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43371 } }
I20260619 16:46:04.753176 12754 leader_election.cc:302] T 00000000000000000000000000000000 P beda6ee725e9415687ecdf160264704b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: beda6ee725e9415687ecdf160264704b; no voters:
I20260619 16:46:04.753883 12754 leader_election.cc:288] T 00000000000000000000000000000000 P beda6ee725e9415687ecdf160264704b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 16:46:04.753995 12759 raft_consensus.cc:2781] T 00000000000000000000000000000000 P beda6ee725e9415687ecdf160264704b [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:46:04.755630 12757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:04.713705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51410 (local address 127.0.97.62:43371)
0619 16:46:04.726249 (+ 12544us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:04.726263 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:04.728156 (+ 1893us) server_negotiation.cc:407] Connection header received
0619 16:46:04.731166 (+ 3010us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:04.731221 (+ 55us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:04.731980 (+ 759us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:04.732625 (+ 645us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:46:04.735319 (+ 2694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:04.737394 (+ 2075us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:04.739482 (+ 2088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:04.740558 (+ 1076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:04.741405 (+ 847us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:04.745829 (+ 4424us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:46:04.745841 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:46:04.745862 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:46:04.745924 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:46:04.749683 (+ 3759us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:04.750631 (+ 948us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:04.750641 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:04.750650 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:04.750869 (+ 219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:04.751343 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:04.751348 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:04.751352 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:04.752109 (+ 757us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:46:04.753480 (+ 1371us) server_negotiation.cc:1035] Waiting for connection context
0619 16:46:04.753788 (+ 308us) server_negotiation.cc:299] Negotiation successful
0619 16:46:04.754211 (+ 423us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12268,"thread_start_us":12063,"threads_started":1}
I20260619 16:46:04.763037 12759 raft_consensus.cc:686] T 00000000000000000000000000000000 P beda6ee725e9415687ecdf160264704b [term 1 LEADER]: Becoming Leader. State: Replica: beda6ee725e9415687ecdf160264704b, State: Running, Role: LEADER
I20260619 16:46:04.763630 12759 consensus_queue.cc:237] T 00000000000000000000000000000000 P beda6ee725e9415687ecdf160264704b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "beda6ee725e9415687ecdf160264704b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43371 } }
I20260619 16:46:04.764092 12754 sys_catalog.cc:564] T 00000000000000000000000000000000 P beda6ee725e9415687ecdf160264704b [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:46:04.768831 12761 sys_catalog.cc:455] T 00000000000000000000000000000000 P beda6ee725e9415687ecdf160264704b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "beda6ee725e9415687ecdf160264704b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "beda6ee725e9415687ecdf160264704b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43371 } } }
I20260619 16:46:04.769127 12760 sys_catalog.cc:455] T 00000000000000000000000000000000 P beda6ee725e9415687ecdf160264704b [sys.catalog]: SysCatalogTable state changed. Reason: New leader beda6ee725e9415687ecdf160264704b. Latest consensus state: current_term: 1 leader_uuid: "beda6ee725e9415687ecdf160264704b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "beda6ee725e9415687ecdf160264704b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43371 } } }
I20260619 16:46:04.770713 12761 sys_catalog.cc:458] T 00000000000000000000000000000000 P beda6ee725e9415687ecdf160264704b [sys.catalog]: This master's current role is: LEADER
I20260619 16:46:04.770747 12760 sys_catalog.cc:458] T 00000000000000000000000000000000 P beda6ee725e9415687ecdf160264704b [sys.catalog]: This master's current role is: LEADER
I20260619 16:46:04.788015 12767 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:46:04.801644 12767 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:46:04.819248 12767 catalog_manager.cc:1347] Generated new cluster ID: 3723525b8cfd49fe8a8b3401b4edf47b
I20260619 16:46:04.819478 12767 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:46:04.824268 12754 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887385903558-388-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260619 16:46:04.877019 12767 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 16:46:04.879132 12767 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:46:04.889935 12767 catalog_manager.cc:5868] T 00000000000000000000000000000000 P beda6ee725e9415687ecdf160264704b: Generated new TSK 0
I20260619 16:46:04.891098 12767 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887385903558-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887385903558-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260619 16:46:05.618841 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu
/tmp/dist-test-taskEs2EH8/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887385903558-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887385903558-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887385903558-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887385903558-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:60203/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43371
--keytab_file=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887385903558-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40517
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887385903558-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887385903558-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887385903558-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 16:46:05 dist-test-slave-wvz7 krb5kdc[12243](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781887565, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260619 16:46:06.012193 12800 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:46:06.012478 12800 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:46:06.012600 12800 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:46:06.023527 12800 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:46:06.023672 12800 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:46:06.023872 12800 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260619 16:46:06.039733 12800 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40517
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887385903558-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887385903558-388-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.97.20:60203/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887385903558-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887385903558-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43371
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887385903558-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:46:06.041457 12800 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:46:06.043215 12800 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:46:06.059248 12810 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:46:06.060674 12811 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:46:06.068521 12813 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:46:06.068864 12800 server_base.cc:1034] running on GCE node
I20260619 16:46:06.069784 12800 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:46:06.071010 12800 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:46:06.072306 12800 hybrid_clock.cc:648] HybridClock initialized: now 1781887566072228 us; error 67 us; skew 500 ppm
I20260619 16:46:06.072674 12800 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:46:06 dist-test-slave-wvz7 krb5kdc[12243](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887566, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:46:06.076797 12800 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260619 16:46:06.078784 12800 webserver.cc:460] Webserver started at http://127.0.97.1:41119/ using document root <none> and password file <none>
I20260619 16:46:06.079681 12800 fs_manager.cc:359] Metadata directory not provided
I20260619 16:46:06.079862 12800 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:46:06.080237 12800 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 16:46:06 dist-test-slave-wvz7 krb5kdc[12243](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887566, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260619 16:46:06.099947 12800 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887385903558-388-0/minicluster-data/ts-0/data/instance:
uuid: "d6b020ec64a941d6ad345b788fa9a341"
format_stamp: "Formatted at 2026-06-19 16:46:06 on dist-test-slave-wvz7"
server_key: "6e75906b13768d208cef39b9bffd2b1f"
server_key_iv: "6cbf4c8756a911a1c292db30bf0ed7ba"
server_key_version: "kuduclusterkey@0"
I20260619 16:46:06.101183 12800 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887385903558-388-0/minicluster-data/ts-0/wal/instance:
uuid: "d6b020ec64a941d6ad345b788fa9a341"
format_stamp: "Formatted at 2026-06-19 16:46:06 on dist-test-slave-wvz7"
server_key: "6e75906b13768d208cef39b9bffd2b1f"
server_key_iv: "6cbf4c8756a911a1c292db30bf0ed7ba"
server_key_version: "kuduclusterkey@0"
I20260619 16:46:06.120371 12800 fs_manager.cc:691] Time spent creating directory manager: real 0.019s user 0.012s sys 0.004s
I20260619 16:46:06.154419 12820 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:46:06.156072 12800 fs_manager.cc:725] Time spent opening block manager: real 0.018s user 0.003s sys 0.015s
I20260619 16:46:06.156258 12800 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887385903558-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887385903558-388-0/minicluster-data/ts-0/wal
uuid: "d6b020ec64a941d6ad345b788fa9a341"
format_stamp: "Formatted at 2026-06-19 16:46:06 on dist-test-slave-wvz7"
server_key: "6e75906b13768d208cef39b9bffd2b1f"
server_key_iv: "6cbf4c8756a911a1c292db30bf0ed7ba"
server_key_version: "kuduclusterkey@0"
I20260619 16:46:06.156478 12800 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887385903558-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887385903558-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887385903558-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:46:06.186111 12800 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:46:06.191411 12800 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:46:06.191784 12800 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:46:06.203943 12800 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:46:06.205516 12800 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:46:06.205600 12800 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:46:06.205761 12800 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:46:06.205822 12800 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:46:06.276602 12800 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:42815
I20260619 16:46:06.276710 12932 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:42815 every 8 connection(s)
I20260619 16:46:06.278337 12800 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887385903558-388-0/minicluster-data/ts-0/data/info.pb
I20260619 16:46:06.288300 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu as pid 12800
I20260619 16:46:06.288509 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887385903558-388-0/minicluster-data/ts-0/wal/instance
I20260619 16:46:06.289326 388 external_mini_cluster.cc:1428] Setting key 445fba41395ca70aa6c5139395d70135
Jun 19 16:46:06 dist-test-slave-wvz7 krb5kdc[12243](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887566, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260619 16:46:06.352352 12935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:06.287883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43371 (local address 127.0.97.1:60719)
0619 16:46:06.309795 (+ 21912us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:06.309860 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0619 16:46:06.312446 (+ 2586us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:46:06.322035 (+ 9589us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:46:06.322051 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:46:06.323995 (+ 1944us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:46:06.324808 (+ 813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:06.324832 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:06.327632 (+ 2800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:06.327638 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:06.329702 (+ 2064us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:06.329726 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:06.330988 (+ 1262us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:06.330996 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:06.331285 (+ 289us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:06.332076 (+ 791us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:46:06.332106 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:46:06.335206 (+ 3100us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:46:06.339171 (+ 3965us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:46:06.339180 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:46:06.339198 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:46:06.339735 (+ 537us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:46:06.340317 (+ 582us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:46:06.340322 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:46:06.340326 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:46:06.348698 (+ 8372us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:46:06.349580 (+ 882us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:46:06.349589 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:46:06.350094 (+ 505us) client_negotiation.cc:769] Sending connection context
0619 16:46:06.350391 (+ 297us) client_negotiation.cc:240] Negotiation successful
0619 16:46:06.351004 (+ 613us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":21520,"thread_start_us":25359,"threads_started":1}
I20260619 16:46:06.353480 12936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:06.288484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:60719 (local address 127.0.97.62:43371)
0619 16:46:06.309027 (+ 20543us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:06.309034 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:06.312947 (+ 3913us) server_negotiation.cc:407] Connection header received
0619 16:46:06.313022 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:06.313028 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:06.313126 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:06.313231 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:46:06.325034 (+ 11803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:06.327424 (+ 2390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:06.329914 (+ 2490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:06.330562 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:06.330689 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:06.335457 (+ 4768us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:46:06.335463 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:46:06.335468 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:46:06.335507 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:46:06.338863 (+ 3356us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:06.339878 (+ 1015us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:06.339884 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:06.339888 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:06.340192 (+ 304us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:06.348895 (+ 8703us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:06.348901 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:06.348905 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:06.349334 (+ 429us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:46:06.349487 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0619 16:46:06.352877 (+ 3390us) server_negotiation.cc:299] Negotiation successful
0619 16:46:06.353131 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20389,"thread_start_us":20562,"threads_started":1}
I20260619 16:46:06.355669 12933 heartbeater.cc:347] Connected to a master server at 127.0.97.62:43371
I20260619 16:46:06.355978 12933 heartbeater.cc:464] Registering TS with master...
I20260619 16:46:06.356768 12933 heartbeater.cc:511] Master 127.0.97.62:43371 requested a full tablet report, sending...
I20260619 16:46:06.359131 12719 ts_manager.cc:194] Registered new tserver with Master: d6b020ec64a941d6ad345b788fa9a341 (127.0.97.1:42815)
I20260619 16:46:06.361117 12719 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:60719
Jun 19 16:46:06 dist-test-slave-wvz7 krb5kdc[12243](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887565, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887385903558-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887385903558-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260619 16:46:06.392261 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu
/tmp/dist-test-taskEs2EH8/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887385903558-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887385903558-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887385903558-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887385903558-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:60203/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43371
--keytab_file=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887385903558-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40517
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887385903558-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887385903558-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887385903558-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 16:46:06 dist-test-slave-wvz7 krb5kdc[12243](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887565, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
W20260619 16:46:06.883935 12939 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:46:06.884400 12939 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:46:06.884699 12939 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:46:06.895892 12939 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:46:06.896059 12939 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:46:06.896292 12939 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260619 16:46:06.911801 12939 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40517
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887385903558-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887385903558-388-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.97.20:60203/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887385903558-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887385903558-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43371
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887385903558-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:46:06.913569 12939 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:46:06.915262 12939 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:46:06.958243 12946 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:46:06.964402 12947 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:46:06.971808 12950 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:46:06.972333 12939 server_base.cc:1034] running on GCE node
I20260619 16:46:06.973289 12939 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:46:06.982028 12939 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:46:06.985908 12939 hybrid_clock.cc:648] HybridClock initialized: now 1781887566985789 us; error 112 us; skew 500 ppm
I20260619 16:46:06.986405 12939 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:46:06 dist-test-slave-wvz7 krb5kdc[12243](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887566, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:46:06.991982 12939 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260619 16:46:07.005493 12939 webserver.cc:460] Webserver started at http://127.0.97.2:41333/ using document root <none> and password file <none>
I20260619 16:46:07.006628 12939 fs_manager.cc:359] Metadata directory not provided
I20260619 16:46:07.006853 12939 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:46:07.007283 12939 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 16:46:07 dist-test-slave-wvz7 krb5kdc[12243](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887566, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260619 16:46:07.043629 12939 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887385903558-388-0/minicluster-data/ts-1/data/instance:
uuid: "d4e75f2cdfcf45a787e5007b8a111a2c"
format_stamp: "Formatted at 2026-06-19 16:46:07 on dist-test-slave-wvz7"
server_key: "6486d8991cc3548a8bca193b7248936d"
server_key_iv: "da85a81cf331cfbe725fd4bcdb1e76fd"
server_key_version: "kuduclusterkey@0"
I20260619 16:46:07.044831 12939 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887385903558-388-0/minicluster-data/ts-1/wal/instance:
uuid: "d4e75f2cdfcf45a787e5007b8a111a2c"
format_stamp: "Formatted at 2026-06-19 16:46:07 on dist-test-slave-wvz7"
server_key: "6486d8991cc3548a8bca193b7248936d"
server_key_iv: "da85a81cf331cfbe725fd4bcdb1e76fd"
server_key_version: "kuduclusterkey@0"
I20260619 16:46:07.069061 12939 fs_manager.cc:691] Time spent creating directory manager: real 0.024s user 0.007s sys 0.004s
I20260619 16:46:07.090950 12963 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:46:07.092469 12939 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260619 16:46:07.092638 12939 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887385903558-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887385903558-388-0/minicluster-data/ts-1/wal
uuid: "d4e75f2cdfcf45a787e5007b8a111a2c"
format_stamp: "Formatted at 2026-06-19 16:46:07 on dist-test-slave-wvz7"
server_key: "6486d8991cc3548a8bca193b7248936d"
server_key_iv: "da85a81cf331cfbe725fd4bcdb1e76fd"
server_key_version: "kuduclusterkey@0"
I20260619 16:46:07.092855 12939 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887385903558-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887385903558-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887385903558-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:46:07.109268 12939 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:46:07.113125 12939 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:46:07.113411 12939 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:46:07.114478 12939 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:46:07.115927 12939 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:46:07.116050 12939 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260619 16:46:07.116205 12939 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:46:07.116261 12939 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:46:07.161855 12939 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:44875
I20260619 16:46:07.161881 13075 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:44875 every 8 connection(s)
I20260619 16:46:07.163597 12939 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887385903558-388-0/minicluster-data/ts-1/data/info.pb
I20260619 16:46:07.168073 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu as pid 12939
I20260619 16:46:07.168222 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887385903558-388-0/minicluster-data/ts-1/wal/instance
I20260619 16:46:07.168756 388 external_mini_cluster.cc:1428] Setting key 4eacf2b336e97ea0a1e033115862b947
Jun 19 16:46:07 dist-test-slave-wvz7 krb5kdc[12243](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887566, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
I20260619 16:46:07.206041 13079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:07.167553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:55787 (local address 127.0.97.62:43371)
0619 16:46:07.183106 (+ 15553us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:07.183113 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:07.186743 (+ 3630us) server_negotiation.cc:407] Connection header received
0619 16:46:07.186817 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:07.186822 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:07.186912 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:07.187014 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:46:07.189772 (+ 2758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:07.190849 (+ 1077us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:07.193066 (+ 2217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:07.193705 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:07.193857 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:07.199205 (+ 5348us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:46:07.199212 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:46:07.199217 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:46:07.199258 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:46:07.202544 (+ 3286us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:07.203387 (+ 843us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:07.203393 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:07.203397 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:07.203551 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:07.204033 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:07.204038 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:07.204042 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:07.204392 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:46:07.204572 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0619 16:46:07.205400 (+ 828us) server_negotiation.cc:299] Negotiation successful
0619 16:46:07.205642 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15396,"thread_start_us":15253,"threads_started":1}
I20260619 16:46:07.207235 13078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:07.167043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43371 (local address 127.0.97.2:55787)
0619 16:46:07.183728 (+ 16685us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:07.183794 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0619 16:46:07.186317 (+ 2523us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:46:07.187472 (+ 1155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:46:07.187496 (+ 24us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:46:07.188111 (+ 615us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:46:07.188975 (+ 864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:07.189000 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:07.191053 (+ 2053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:07.191059 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:07.192890 (+ 1831us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:07.192912 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:07.193935 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:07.193941 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:07.194196 (+ 255us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:07.194978 (+ 782us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:46:07.195006 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:46:07.198907 (+ 3901us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:46:07.202689 (+ 3782us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:46:07.202699 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:46:07.202716 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:46:07.203256 (+ 540us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:46:07.203666 (+ 410us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:46:07.203672 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:46:07.203675 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:46:07.203929 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:46:07.204564 (+ 635us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:46:07.204572 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:46:07.204979 (+ 407us) client_negotiation.cc:769] Sending connection context
0619 16:46:07.205283 (+ 304us) client_negotiation.cc:240] Negotiation successful
0619 16:46:07.205883 (+ 600us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16365,"thread_start_us":16140,"threads_started":1}
I20260619 16:46:07.209100 13076 heartbeater.cc:347] Connected to a master server at 127.0.97.62:43371
I20260619 16:46:07.209416 13076 heartbeater.cc:464] Registering TS with master...
I20260619 16:46:07.210147 13076 heartbeater.cc:511] Master 127.0.97.62:43371 requested a full tablet report, sending...
I20260619 16:46:07.211692 12719 ts_manager.cc:194] Registered new tserver with Master: d4e75f2cdfcf45a787e5007b8a111a2c (127.0.97.2:44875)
I20260619 16:46:07.212808 12719 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:55787
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887385903558-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887385903558-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260619 16:46:07.229010 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu
/tmp/dist-test-taskEs2EH8/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887385903558-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887385903558-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887385903558-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887385903558-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:60203/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43371
--keytab_file=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887385903558-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40517
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887385903558-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887385903558-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887385903558-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 16:46:07.365152 12933 heartbeater.cc:503] Master 127.0.97.62:43371 was elected leader, sending a full tablet report...
W20260619 16:46:07.537823 13082 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:46:07.538090 13082 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:46:07.538183 13082 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:46:07.545199 13082 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:46:07.545310 13082 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:46:07.545418 13082 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260619 16:46:07.554915 13082 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40517
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887385903558-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887385903558-388-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.97.20:60203/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887385903558-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887385903558-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43371
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887385903558-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:46:07.556097 13082 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:46:07.557289 13082 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:46:07.569227 13088 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:46:07.569236 13090 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:46:07.569236 13087 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:46:07.569998 13082 server_base.cc:1034] running on GCE node
I20260619 16:46:07.570521 13082 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:46:07.571410 13082 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:46:07.572593 13082 hybrid_clock.cc:648] HybridClock initialized: now 1781887567572546 us; error 40 us; skew 500 ppm
I20260619 16:46:07.572899 13082 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:46:07 dist-test-slave-wvz7 krb5kdc[12243](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887567, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:46:07.575762 13082 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260619 16:46:07.577147 13082 webserver.cc:460] Webserver started at http://127.0.97.3:35899/ using document root <none> and password file <none>
I20260619 16:46:07.577816 13082 fs_manager.cc:359] Metadata directory not provided
I20260619 16:46:07.577912 13082 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:46:07.578150 13082 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 16:46:07 dist-test-slave-wvz7 krb5kdc[12243](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887567, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260619 16:46:07.591529 13082 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887385903558-388-0/minicluster-data/ts-2/data/instance:
uuid: "0e27e43e6d214571870edb0b771dbf91"
format_stamp: "Formatted at 2026-06-19 16:46:07 on dist-test-slave-wvz7"
server_key: "95556518c52c26c0a767d83da754297c"
server_key_iv: "e1d5c00418eae2f151ab74cbe4264377"
server_key_version: "kuduclusterkey@0"
I20260619 16:46:07.592296 13082 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887385903558-388-0/minicluster-data/ts-2/wal/instance:
uuid: "0e27e43e6d214571870edb0b771dbf91"
format_stamp: "Formatted at 2026-06-19 16:46:07 on dist-test-slave-wvz7"
server_key: "95556518c52c26c0a767d83da754297c"
server_key_iv: "e1d5c00418eae2f151ab74cbe4264377"
server_key_version: "kuduclusterkey@0"
I20260619 16:46:07.597818 13082 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260619 16:46:07.610942 13097 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:46:07.612534 13082 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260619 16:46:07.612687 13082 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887385903558-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887385903558-388-0/minicluster-data/ts-2/wal
uuid: "0e27e43e6d214571870edb0b771dbf91"
format_stamp: "Formatted at 2026-06-19 16:46:07 on dist-test-slave-wvz7"
server_key: "95556518c52c26c0a767d83da754297c"
server_key_iv: "e1d5c00418eae2f151ab74cbe4264377"
server_key_version: "kuduclusterkey@0"
I20260619 16:46:07.612886 13082 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887385903558-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887385903558-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887385903558-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:46:07.638332 13082 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:46:07.641876 13082 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:46:07.642145 13082 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:46:07.643054 13082 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:46:07.644304 13082 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:46:07.644377 13082 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:46:07.644487 13082 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:46:07.644536 13082 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:46:07.679239 13082 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:35299
I20260619 16:46:07.679286 13209 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:35299 every 8 connection(s)
I20260619 16:46:07.680411 13082 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887385903558-388-0/minicluster-data/ts-2/data/info.pb
I20260619 16:46:07.689180 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu as pid 13082
I20260619 16:46:07.689334 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887385903558-388-0/minicluster-data/ts-2/wal/instance
I20260619 16:46:07.689778 388 external_mini_cluster.cc:1428] Setting key bf7f4f32ef060cea8d4df2178d7e0356
Jun 19 16:46:07 dist-test-slave-wvz7 krb5kdc[12243](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887567, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 16:46:07.698724 13079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:07.682743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:50211 (local address 127.0.97.62:43371)
0619 16:46:07.682913 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:07.682918 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:07.683927 (+ 1009us) server_negotiation.cc:407] Connection header received
0619 16:46:07.685612 (+ 1685us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:07.685618 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:07.685709 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:07.685828 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:46:07.687138 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:07.687845 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:07.689347 (+ 1502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:07.689787 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:07.689904 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:07.693499 (+ 3595us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:46:07.693504 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:46:07.693507 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:46:07.693546 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:46:07.695851 (+ 2305us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:07.696630 (+ 779us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:07.696636 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:07.696640 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:07.696760 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:07.697143 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:07.697148 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:07.697151 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:07.697442 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:46:07.697570 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0619 16:46:07.698204 (+ 634us) server_negotiation.cc:299] Negotiation successful
0619 16:46:07.698423 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260619 16:46:07.699564 13212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:07.683136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43371 (local address 127.0.97.3:50211)
0619 16:46:07.683783 (+ 647us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:07.683819 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0619 16:46:07.685353 (+ 1534us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:46:07.686033 (+ 680us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:46:07.686042 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:46:07.686441 (+ 399us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:46:07.686980 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:07.686996 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:07.687989 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:07.687992 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:07.689178 (+ 1186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:07.689189 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:07.690016 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:07.690024 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:07.690313 (+ 289us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:07.690915 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:46:07.690958 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:46:07.693297 (+ 2339us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:46:07.695976 (+ 2679us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:46:07.695984 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:46:07.695998 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:46:07.696506 (+ 508us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:46:07.696873 (+ 367us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:46:07.696877 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:46:07.696879 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:46:07.697043 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:46:07.697557 (+ 514us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:46:07.697564 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:46:07.697890 (+ 326us) client_negotiation.cc:769] Sending connection context
0619 16:46:07.698116 (+ 226us) client_negotiation.cc:240] Negotiation successful
0619 16:46:07.698568 (+ 452us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":437,"thread_start_us":257,"threads_started":1}
I20260619 16:46:07.700708 13210 heartbeater.cc:347] Connected to a master server at 127.0.97.62:43371
I20260619 16:46:07.700928 13210 heartbeater.cc:464] Registering TS with master...
I20260619 16:46:07.701524 13210 heartbeater.cc:511] Master 127.0.97.62:43371 requested a full tablet report, sending...
I20260619 16:46:07.702800 12719 ts_manager.cc:194] Registered new tserver with Master: 0e27e43e6d214571870edb0b771dbf91 (127.0.97.3:35299)
I20260619 16:46:07.703608 12719 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:50211
I20260619 16:46:07.705121 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 16:46:07.706733 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu pid 12800
Jun 19 16:46:07 dist-test-slave-wvz7 krb5kdc[12243](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887564, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260619 16:46:07.723592 13214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:07.707713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38048 (local address 127.0.97.1:42815)
0619 16:46:07.708358 (+ 645us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:07.708366 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:07.708392 (+ 26us) server_negotiation.cc:407] Connection header received
0619 16:46:07.708451 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:07.708458 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:07.708669 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:07.708819 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:46:07.710287 (+ 1468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:07.711766 (+ 1479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:07.713209 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:07.714030 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:07.714143 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:07.718292 (+ 4149us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:46:07.718303 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:46:07.718315 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:46:07.718346 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:46:07.721057 (+ 2711us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:07.721680 (+ 623us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:07.721686 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:07.721689 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:07.721895 (+ 206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:07.722239 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:07.722242 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:07.722244 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:07.722566 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:46:07.722803 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0619 16:46:07.723008 (+ 205us) server_negotiation.cc:299] Negotiation successful
0619 16:46:07.723205 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":493,"thread_start_us":342,"threads_started":1}
I20260619 16:46:07.724792 12928 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:46:07.804311 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu with pid 12800
I20260619 16:46:07.816133 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu pid 12939
Jun 19 16:46:07 dist-test-slave-wvz7 krb5kdc[12243](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887564, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260619 16:46:07.831878 13216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:07.817058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48564 (local address 127.0.97.2:44875)
0619 16:46:07.817649 (+ 591us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:07.817656 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:07.817680 (+ 24us) server_negotiation.cc:407] Connection header received
0619 16:46:07.817753 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:07.817762 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:07.817987 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:07.818110 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:46:07.819440 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:07.820399 (+ 959us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:07.821719 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:07.822309 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:07.822409 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:07.826637 (+ 4228us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:46:07.826648 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:46:07.826662 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:46:07.826696 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:46:07.829228 (+ 2532us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:07.829988 (+ 760us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:07.829996 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:07.829998 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:07.830152 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:07.830513 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:07.830517 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:07.830519 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:07.830894 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:46:07.831115 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0619 16:46:07.831324 (+ 209us) server_negotiation.cc:299] Negotiation successful
0619 16:46:07.831522 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":429,"thread_start_us":271,"threads_started":1}
I20260619 16:46:07.833062 13071 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:46:07.915458 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu with pid 12939
I20260619 16:46:07.927194 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu pid 13082
Jun 19 16:46:07 dist-test-slave-wvz7 krb5kdc[12243](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887564, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260619 16:46:07.942426 13218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:07.927968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57340 (local address 127.0.97.3:35299)
0619 16:46:07.928548 (+ 580us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:07.928555 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:07.928584 (+ 29us) server_negotiation.cc:407] Connection header received
0619 16:46:07.928642 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:07.928648 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:07.928853 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:07.928975 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:46:07.930268 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:07.931278 (+ 1010us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:07.932735 (+ 1457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:07.933282 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:07.933378 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:07.937579 (+ 4201us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:46:07.937590 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:46:07.937608 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:46:07.937643 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:46:07.940067 (+ 2424us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:07.940705 (+ 638us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:07.940711 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:07.940713 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:07.940827 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:07.941170 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:07.941173 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:07.941175 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:07.941472 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:46:07.941717 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0619 16:46:07.941963 (+ 246us) server_negotiation.cc:299] Negotiation successful
0619 16:46:07.942155 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":441,"thread_start_us":286,"threads_started":1}
I20260619 16:46:07.943605 13205 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:46:08.025522 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu with pid 13082
I20260619 16:46:08.037050 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu pid 12687
I20260619 16:46:08.038259 12749 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:46:08.145028 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu with pid 12687
I20260619 16:46:08.164356 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu
/tmp/dist-test-taskEs2EH8/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887385903558-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887385903558-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887385903558-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887385903558-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:43371
--webserver_interface=127.0.97.62
--webserver_port=34833
--keytab_file=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887385903558-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40517
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:43371
--ranger_config_path=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887385903558-388-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:60203/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:60203/kms with env {KRB5CCNAME=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887385903558-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887385903558-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887385903558-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:46:08.469760 13221 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:46:08.470055 13221 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:46:08.470181 13221 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:46:08.477249 13221 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:46:08.477365 13221 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:46:08.477445 13221 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:46:08.477520 13221 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:46:08.477586 13221 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:46:08.487316 13221 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40517
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887385903558-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887385903558-388-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:60203/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:43371
--ranger_config_path=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887385903558-388-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887385903558-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:43371
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887385903558-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=34833
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887385903558-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:46:08.488536 13221 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:46:08.489823 13221 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:46:08.500468 13229 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:46:08.500552 13226 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:46:08.500510 13227 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:46:08.501434 13221 server_base.cc:1034] running on GCE node
I20260619 16:46:08.502171 13221 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:46:08.503463 13221 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:46:08.504649 13221 hybrid_clock.cc:648] HybridClock initialized: now 1781887568504603 us; error 46 us; skew 500 ppm
I20260619 16:46:08.504954 13221 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:46:08 dist-test-slave-wvz7 krb5kdc[12243](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887568, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:46:08.507882 13221 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260619 16:46:08.509199 13221 webserver.cc:460] Webserver started at http://127.0.97.62:34833/ using document root <none> and password file <none>
I20260619 16:46:08.509815 13221 fs_manager.cc:359] Metadata directory not provided
I20260619 16:46:08.509933 13221 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 19 16:46:08 dist-test-slave-wvz7 krb5kdc[12243](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887568, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260619 16:46:08.557397 13221 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.004s sys 0.000s
I20260619 16:46:08.560854 13237 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:46:08.562620 13221 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 16:46:08.562808 13221 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887385903558-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887385903558-388-0/minicluster-data/master-0/wal
uuid: "beda6ee725e9415687ecdf160264704b"
format_stamp: "Formatted at 2026-06-19 16:46:04 on dist-test-slave-wvz7"
server_key: "6c7e90a8d42aeafd13c5e11cc6e00071"
server_key_iv: "15ac713d0d631840c0ef1a940b474e91"
server_key_version: "kuduclusterkey@0"
I20260619 16:46:08.564131 13221 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887385903558-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887385903558-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887385903558-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:46:08.580571 13221 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:46:08.584015 13221 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:46:08.584239 13221 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:46:08.600929 13221 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:43371
I20260619 16:46:08.600958 13288 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:43371 every 8 connection(s)
I20260619 16:46:08.602312 13221 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887385903558-388-0/minicluster-data/master-0/data/info.pb
I20260619 16:46:08.603928 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu as pid 13221
I20260619 16:46:08.604885 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu
/tmp/dist-test-taskEs2EH8/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887385903558-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887385903558-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887385903558-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887385903558-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:60203/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:42815
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:43371
--webserver_port=41119
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887385903558-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40517
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:60203/kms with env {KRB5CCNAME=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887385903558-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887385903558-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887385903558-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 16:46:08.611178 13289 sys_catalog.cc:263] Verifying existing consensus state
I20260619 16:46:08.614828 13289 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P beda6ee725e9415687ecdf160264704b: Bootstrap starting.
I20260619 16:46:08.632714 13289 log.cc:824] T 00000000000000000000000000000000 P beda6ee725e9415687ecdf160264704b: Log is configured to *not* fsync() on all Append() calls
I20260619 16:46:08.640897 13289 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P beda6ee725e9415687ecdf160264704b: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 16:46:08.641361 13289 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P beda6ee725e9415687ecdf160264704b: Bootstrap complete.
I20260619 16:46:08.649602 13289 raft_consensus.cc:348] T 00000000000000000000000000000000 P beda6ee725e9415687ecdf160264704b [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "beda6ee725e9415687ecdf160264704b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43371 } }
I20260619 16:46:08.650437 13289 raft_consensus.cc:729] T 00000000000000000000000000000000 P beda6ee725e9415687ecdf160264704b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: beda6ee725e9415687ecdf160264704b, State: Initialized, Role: FOLLOWER
I20260619 16:46:08.650987 13289 consensus_queue.cc:260] T 00000000000000000000000000000000 P beda6ee725e9415687ecdf160264704b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "beda6ee725e9415687ecdf160264704b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43371 } }
I20260619 16:46:08.651190 13289 raft_consensus.cc:388] T 00000000000000000000000000000000 P beda6ee725e9415687ecdf160264704b [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:46:08.651290 13289 raft_consensus.cc:482] T 00000000000000000000000000000000 P beda6ee725e9415687ecdf160264704b [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:46:08.651428 13289 raft_consensus.cc:3037] T 00000000000000000000000000000000 P beda6ee725e9415687ecdf160264704b [term 1 FOLLOWER]: Advancing to term 2
I20260619 16:46:08.654937 13289 raft_consensus.cc:504] T 00000000000000000000000000000000 P beda6ee725e9415687ecdf160264704b [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "beda6ee725e9415687ecdf160264704b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43371 } }
I20260619 16:46:08.655380 13289 leader_election.cc:302] T 00000000000000000000000000000000 P beda6ee725e9415687ecdf160264704b [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: beda6ee725e9415687ecdf160264704b; no voters:
I20260619 16:46:08.655895 13289 leader_election.cc:288] T 00000000000000000000000000000000 P beda6ee725e9415687ecdf160264704b [CANDIDATE]: Term 2 election: Requested vote from peers
I20260619 16:46:08.656035 13293 raft_consensus.cc:2781] T 00000000000000000000000000000000 P beda6ee725e9415687ecdf160264704b [term 2 FOLLOWER]: Leader election won for term 2
I20260619 16:46:08.657024 13293 raft_consensus.cc:686] T 00000000000000000000000000000000 P beda6ee725e9415687ecdf160264704b [term 2 LEADER]: Becoming Leader. State: Replica: beda6ee725e9415687ecdf160264704b, State: Running, Role: LEADER
I20260619 16:46:08.657626 13293 consensus_queue.cc:237] T 00000000000000000000000000000000 P beda6ee725e9415687ecdf160264704b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "beda6ee725e9415687ecdf160264704b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43371 } }
I20260619 16:46:08.657871 13289 sys_catalog.cc:564] T 00000000000000000000000000000000 P beda6ee725e9415687ecdf160264704b [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:46:08.659832 13295 sys_catalog.cc:455] T 00000000000000000000000000000000 P beda6ee725e9415687ecdf160264704b [sys.catalog]: SysCatalogTable state changed. Reason: New leader beda6ee725e9415687ecdf160264704b. Latest consensus state: current_term: 2 leader_uuid: "beda6ee725e9415687ecdf160264704b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "beda6ee725e9415687ecdf160264704b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43371 } } }
I20260619 16:46:08.660190 13295 sys_catalog.cc:458] T 00000000000000000000000000000000 P beda6ee725e9415687ecdf160264704b [sys.catalog]: This master's current role is: LEADER
I20260619 16:46:08.660679 13294 sys_catalog.cc:455] T 00000000000000000000000000000000 P beda6ee725e9415687ecdf160264704b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "beda6ee725e9415687ecdf160264704b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "beda6ee725e9415687ecdf160264704b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43371 } } }
I20260619 16:46:08.660965 13294 sys_catalog.cc:458] T 00000000000000000000000000000000 P beda6ee725e9415687ecdf160264704b [sys.catalog]: This master's current role is: LEADER
I20260619 16:46:08.671197 13299 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:46:08.677237 13299 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:46:08.678596 13299 catalog_manager.cc:1259] Loaded cluster ID: 3723525b8cfd49fe8a8b3401b4edf47b
I20260619 16:46:08.678742 13299 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:46:08.682691 13299 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:46:08.684051 13299 catalog_manager.cc:5879] T 00000000000000000000000000000000 P beda6ee725e9415687ecdf160264704b: Loaded TSK: 0
I20260619 16:46:08.685683 13299 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 16:46:08.687942 13289 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887385903558-388-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
W20260619 16:46:09.039418 13291 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:46:09.039835 13291 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:46:09.040041 13291 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:46:09.051400 13291 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:46:09.051586 13291 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:46:09.051803 13291 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260619 16:46:09.066860 13291 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40517
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887385903558-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887385903558-388-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:60203/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887385903558-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:42815
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887385903558-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=41119
--tserver_master_addrs=127.0.97.62:43371
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887385903558-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:46:09.068550 13291 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:46:09.070290 13291 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:46:09.093238 13326 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:46:09.094602 13327 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:46:09.106025 13329 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:46:09.106668 13291 server_base.cc:1034] running on GCE node
I20260619 16:46:09.107504 13291 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:46:09.112003 13291 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:46:09.113324 13291 hybrid_clock.cc:648] HybridClock initialized: now 1781887569113275 us; error 40 us; skew 500 ppm
I20260619 16:46:09.113770 13291 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:46:09 dist-test-slave-wvz7 krb5kdc[12243](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887569, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:46:09.117923 13291 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260619 16:46:09.119771 13291 webserver.cc:460] Webserver started at http://127.0.97.1:41119/ using document root <none> and password file <none>
I20260619 16:46:09.120747 13291 fs_manager.cc:359] Metadata directory not provided
I20260619 16:46:09.120911 13291 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 19 16:46:09 dist-test-slave-wvz7 krb5kdc[12243](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887569, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260619 16:46:09.149092 13291 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.004s sys 0.000s
I20260619 16:46:09.154736 13337 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:46:09.156546 13291 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.005s sys 0.000s
I20260619 16:46:09.156792 13291 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887385903558-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887385903558-388-0/minicluster-data/ts-0/wal
uuid: "d6b020ec64a941d6ad345b788fa9a341"
format_stamp: "Formatted at 2026-06-19 16:46:06 on dist-test-slave-wvz7"
server_key: "6e75906b13768d208cef39b9bffd2b1f"
server_key_iv: "6cbf4c8756a911a1c292db30bf0ed7ba"
server_key_version: "kuduclusterkey@0"
I20260619 16:46:09.158200 13291 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887385903558-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887385903558-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887385903558-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:46:09.205263 13291 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:46:09.210626 13291 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:46:09.211000 13291 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:46:09.214954 13291 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:46:09.216446 13291 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:46:09.216542 13291 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:46:09.216643 13291 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:46:09.216717 13291 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:46:09.279170 13291 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:42815
I20260619 16:46:09.279222 13450 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:42815 every 8 connection(s)
I20260619 16:46:09.280966 13291 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887385903558-388-0/minicluster-data/ts-0/data/info.pb
I20260619 16:46:09.283313 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu as pid 13291
I20260619 16:46:09.284205 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu
/tmp/dist-test-taskEs2EH8/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887385903558-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887385903558-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887385903558-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887385903558-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:60203/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:44875
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:43371
--webserver_port=41333
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887385903558-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40517
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:60203/kms with env {KRB5CCNAME=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887385903558-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887385903558-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887385903558-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 16:46:09 dist-test-slave-wvz7 krb5kdc[12243](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887569, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 16:46:09.337437 13455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:09.294217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43371 (local address 127.0.97.1:58851)
0619 16:46:09.313376 (+ 19159us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:09.313438 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0619 16:46:09.320337 (+ 6899us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:46:09.322239 (+ 1902us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:46:09.322250 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:46:09.322795 (+ 545us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:46:09.323411 (+ 616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:09.323428 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:09.324990 (+ 1562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:09.324993 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:09.326280 (+ 1287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:09.326292 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:09.327252 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:09.327259 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:09.327503 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:09.328153 (+ 650us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:46:09.328175 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:46:09.330074 (+ 1899us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:46:09.333030 (+ 2956us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:46:09.333038 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:46:09.333052 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:46:09.333690 (+ 638us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:46:09.334265 (+ 575us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:46:09.334270 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:46:09.334274 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:46:09.334448 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:46:09.335175 (+ 727us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:46:09.335183 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:46:09.335462 (+ 279us) client_negotiation.cc:769] Sending connection context
0619 16:46:09.335675 (+ 213us) client_negotiation.cc:240] Negotiation successful
0619 16:46:09.336180 (+ 505us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18853,"thread_start_us":18634,"threads_started":1}
I20260619 16:46:09.337427 13454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:09.294030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:58851 (local address 127.0.97.62:43371)
0619 16:46:09.311039 (+ 17009us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:09.311064 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:09.317872 (+ 6808us) server_negotiation.cc:407] Connection header received
0619 16:46:09.321102 (+ 3230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:09.321117 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:09.321652 (+ 535us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:09.322079 (+ 427us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:46:09.323607 (+ 1528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:09.324843 (+ 1236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:09.326409 (+ 1566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:09.327036 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:09.327214 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:09.330308 (+ 3094us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:46:09.330315 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:46:09.330328 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:46:09.330364 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:46:09.332903 (+ 2539us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:09.333904 (+ 1001us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:09.333913 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:09.333920 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:09.334110 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:09.334563 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:09.334566 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:09.334568 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:09.335039 (+ 471us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:46:09.335210 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0619 16:46:09.335923 (+ 713us) server_negotiation.cc:299] Negotiation successful
0619 16:46:09.336270 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16673,"thread_start_us":16415,"threads_started":1}
I20260619 16:46:09.340513 13451 heartbeater.cc:347] Connected to a master server at 127.0.97.62:43371
I20260619 16:46:09.340818 13451 heartbeater.cc:464] Registering TS with master...
I20260619 16:46:09.341516 13451 heartbeater.cc:511] Master 127.0.97.62:43371 requested a full tablet report, sending...
W20260619 16:46:09.344833 13451 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:43371 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260619 16:46:09.345046 13451 heartbeater.cc:464] Registering TS with master...
I20260619 16:46:09.345322 13451 heartbeater.cc:511] Master 127.0.97.62:43371 requested a full tablet report, sending...
I20260619 16:46:09.346495 13451 heartbeater.cc:464] Registering TS with master...
I20260619 16:46:09.346873 13451 heartbeater.cc:511] Master 127.0.97.62:43371 requested a full tablet report, sending...
W20260619 16:46:09.347955 13451 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260619 16:46:09.682842 13453 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:46:09.683106 13453 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:46:09.683246 13453 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:46:09.690837 13453 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:46:09.691004 13453 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:46:09.691202 13453 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260619 16:46:09.701324 13453 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40517
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887385903558-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887385903558-388-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:60203/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887385903558-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:44875
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887385903558-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=41333
--tserver_master_addrs=127.0.97.62:43371
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887385903558-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:46:09.702594 13453 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:46:09.703828 13453 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:46:09.719341 13469 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:46:09.719372 13470 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:46:09.725261 13472 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:46:09.725262 13453 server_base.cc:1034] running on GCE node
I20260619 16:46:09.725979 13453 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:46:09.726900 13453 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:46:09.728106 13453 hybrid_clock.cc:648] HybridClock initialized: now 1781887569728036 us; error 65 us; skew 500 ppm
I20260619 16:46:09.728430 13453 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:46:09 dist-test-slave-wvz7 krb5kdc[12243](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887569, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:46:09.734169 13453 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260619 16:46:09.736394 13453 webserver.cc:460] Webserver started at http://127.0.97.2:41333/ using document root <none> and password file <none>
I20260619 16:46:09.737061 13453 fs_manager.cc:359] Metadata directory not provided
I20260619 16:46:09.737191 13453 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 19 16:46:09 dist-test-slave-wvz7 krb5kdc[12243](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887569, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260619 16:46:09.774121 13453 fs_manager.cc:709] Time spent opening directory manager: real 0.008s user 0.008s sys 0.000s
I20260619 16:46:09.776947 13481 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:46:09.778427 13453 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 16:46:09.778576 13453 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887385903558-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887385903558-388-0/minicluster-data/ts-1/wal
uuid: "d4e75f2cdfcf45a787e5007b8a111a2c"
format_stamp: "Formatted at 2026-06-19 16:46:07 on dist-test-slave-wvz7"
server_key: "6486d8991cc3548a8bca193b7248936d"
server_key_iv: "da85a81cf331cfbe725fd4bcdb1e76fd"
server_key_version: "kuduclusterkey@0"
I20260619 16:46:09.779498 13453 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887385903558-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887385903558-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887385903558-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:46:09.828248 13453 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:46:09.832054 13453 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:46:09.832294 13453 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:46:09.833245 13453 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:46:09.834700 13453 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:46:09.834791 13453 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:46:09.834892 13453 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:46:09.834957 13453 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 16:46:09 dist-test-slave-wvz7 krb5kdc[12243](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781887569, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:46:09.877419 13453 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:44875
I20260619 16:46:09.877498 13594 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:44875 every 8 connection(s)
I20260619 16:46:09.879164 13453 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887385903558-388-0/minicluster-data/ts-1/data/info.pb
I20260619 16:46:09.882711 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu as pid 13453
I20260619 16:46:09.883461 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu
/tmp/dist-test-taskEs2EH8/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887385903558-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887385903558-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887385903558-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887385903558-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:60203/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:35299
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:43371
--webserver_port=35899
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887385903558-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40517
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:60203/kms with env {KRB5CCNAME=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887385903558-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887385903558-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887385903558-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 16:46:09 dist-test-slave-wvz7 krb5kdc[12243](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887569, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 16:46:09.927917 13598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:09.896348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:33765 (local address 127.0.97.62:43371)
0619 16:46:09.910050 (+ 13702us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:09.910057 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:09.910081 (+ 24us) server_negotiation.cc:407] Connection header received
0619 16:46:09.910453 (+ 372us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:09.910458 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:09.910542 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:09.910644 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:46:09.913358 (+ 2714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:09.914548 (+ 1190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:09.916983 (+ 2435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:09.917601 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:09.917769 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:09.922060 (+ 4291us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:46:09.922066 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:46:09.922072 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:46:09.922112 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:46:09.925121 (+ 3009us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:09.925790 (+ 669us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:09.925795 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:09.925797 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:09.925929 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:09.926320 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:09.926324 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:09.926326 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:09.926571 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:46:09.926697 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0619 16:46:09.927355 (+ 658us) server_negotiation.cc:299] Negotiation successful
0619 16:46:09.927564 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13575,"thread_start_us":13733,"threads_started":1}
I20260619 16:46:09.928927 13597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:09.895737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43371 (local address 127.0.97.2:33765)
0619 16:46:09.907082 (+ 11345us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:09.907131 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0619 16:46:09.909571 (+ 2440us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:46:09.911084 (+ 1513us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:46:09.911096 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:46:09.912373 (+ 1277us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:46:09.913132 (+ 759us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:09.913156 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:09.914782 (+ 1626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:09.914788 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:09.916808 (+ 2020us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:09.916833 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:09.918091 (+ 1258us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:09.918097 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:09.918383 (+ 286us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:09.919075 (+ 692us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:46:09.919106 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:46:09.921814 (+ 2708us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:46:09.925250 (+ 3436us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:46:09.925260 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:46:09.925276 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:46:09.925663 (+ 387us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:46:09.926029 (+ 366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:46:09.926034 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:46:09.926038 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:46:09.926220 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:46:09.926696 (+ 476us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:46:09.926703 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:46:09.927012 (+ 309us) client_negotiation.cc:769] Sending connection context
0619 16:46:09.927251 (+ 239us) client_negotiation.cc:240] Negotiation successful
0619 16:46:09.927765 (+ 514us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11047,"thread_start_us":10825,"threads_started":1}
I20260619 16:46:09.930342 13595 heartbeater.cc:347] Connected to a master server at 127.0.97.62:43371
I20260619 16:46:09.930612 13595 heartbeater.cc:464] Registering TS with master...
I20260619 16:46:09.931267 13595 heartbeater.cc:511] Master 127.0.97.62:43371 requested a full tablet report, sending...
I20260619 16:46:09.932857 13254 ts_manager.cc:194] Registered new tserver with Master: d4e75f2cdfcf45a787e5007b8a111a2c (127.0.97.2:44875)
I20260619 16:46:09.934396 13254 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:33765
W20260619 16:46:10.347425 13599 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:46:10.347815 13599 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:46:10.348011 13599 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260619 16:46:10.348104 13451 heartbeater.cc:464] Registering TS with master...
Jun 19 16:46:10 dist-test-slave-wvz7 krb5kdc[12243](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887569, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260619 16:46:10.348482 13451 heartbeater.cc:511] Master 127.0.97.62:43371 requested a full tablet report, sending...
I20260619 16:46:10.349582 13254 ts_manager.cc:194] Registered new tserver with Master: d6b020ec64a941d6ad345b788fa9a341 (127.0.97.1:42815)
I20260619 16:46:10.350816 13254 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:58851
W20260619 16:46:10.357470 13599 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:46:10.357581 13599 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:46:10.357827 13599 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260619 16:46:10.372427 13599 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40517
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887385903558-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887385903558-388-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:60203/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887385903558-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:35299
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887385903558-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=35899
--tserver_master_addrs=127.0.97.62:43371
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887385903558-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:46:10.374405 13599 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:46:10.376150 13599 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:46:10.402580 13606 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:46:10.412824 13609 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:46:10.416301 13599 server_base.cc:1034] running on GCE node
W20260619 16:46:10.418021 13605 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:46:10.418869 13599 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:46:10.430065 13599 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:46:10.431372 13599 hybrid_clock.cc:648] HybridClock initialized: now 1781887570431294 us; error 78 us; skew 500 ppm
I20260619 16:46:10.431833 13599 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:46:10 dist-test-slave-wvz7 krb5kdc[12243](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887570, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:46:10.435953 13599 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260619 16:46:10.456668 13599 webserver.cc:460] Webserver started at http://127.0.97.3:35899/ using document root <none> and password file <none>
I20260619 16:46:10.457765 13599 fs_manager.cc:359] Metadata directory not provided
I20260619 16:46:10.458006 13599 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 19 16:46:10 dist-test-slave-wvz7 krb5kdc[12243](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887570, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260619 16:46:10.505280 13599 fs_manager.cc:709] Time spent opening directory manager: real 0.013s user 0.004s sys 0.008s
I20260619 16:46:10.530642 13616 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:46:10.532730 13599 fs_manager.cc:725] Time spent opening block manager: real 0.025s user 0.004s sys 0.010s
I20260619 16:46:10.532995 13599 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887385903558-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887385903558-388-0/minicluster-data/ts-2/wal
uuid: "0e27e43e6d214571870edb0b771dbf91"
format_stamp: "Formatted at 2026-06-19 16:46:07 on dist-test-slave-wvz7"
server_key: "95556518c52c26c0a767d83da754297c"
server_key_iv: "e1d5c00418eae2f151ab74cbe4264377"
server_key_version: "kuduclusterkey@0"
I20260619 16:46:10.534515 13599 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887385903558-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887385903558-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887385903558-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:46:10.572714 13599 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:46:10.577809 13599 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:46:10.578145 13599 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:46:10.585453 13599 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:46:10.593629 13599 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:46:10.593832 13599 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260619 16:46:10.593986 13599 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:46:10.594092 13599 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 16:46:10 dist-test-slave-wvz7 krb5kdc[12243](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887569, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260619 16:46:10.896693 13599 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:35299
I20260619 16:46:10.896741 13729 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:35299 every 8 connection(s)
I20260619 16:46:10.898149 13599 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887385903558-388-0/minicluster-data/ts-2/data/info.pb
I20260619 16:46:10.902587 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu as pid 13599
Jun 19 16:46:10 dist-test-slave-wvz7 krb5kdc[12243](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887570, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 16:46:10.938019 13595 heartbeater.cc:503] Master 127.0.97.62:43371 was elected leader, sending a full tablet report...
I20260619 16:46:10.946689 13735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:10.911321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51426 (local address 127.0.97.62:43371)
0619 16:46:10.919952 (+ 8631us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:10.919958 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:10.919977 (+ 19us) server_negotiation.cc:407] Connection header received
0619 16:46:10.920030 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:10.920035 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:10.920099 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:10.920206 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:46:10.925852 (+ 5646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:10.927105 (+ 1253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:10.929269 (+ 2164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:10.929930 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:10.930059 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:10.933464 (+ 3405us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:46:10.933470 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:46:10.933475 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:46:10.933513 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:46:10.940551 (+ 7038us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:10.942130 (+ 1579us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:10.942135 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:10.942138 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:10.942290 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:10.942941 (+ 651us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:10.942945 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:10.942949 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:10.943300 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:46:10.945967 (+ 2667us) server_negotiation.cc:1035] Waiting for connection context
0619 16:46:10.946093 (+ 126us) server_negotiation.cc:299] Negotiation successful
0619 16:46:10.946316 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8510,"thread_start_us":9061,"threads_started":1}
I20260619 16:46:10.948644 13733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:10.903303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:40583 (local address 127.0.97.62:43371)
0619 16:46:10.911390 (+ 8087us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:10.911396 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:10.916432 (+ 5036us) server_negotiation.cc:407] Connection header received
0619 16:46:10.916504 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:10.916510 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:10.916590 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:10.916695 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:46:10.920143 (+ 3448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:10.921282 (+ 1139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:10.925807 (+ 4525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:10.926525 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:10.926616 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:10.936742 (+ 10126us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:46:10.936750 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:46:10.936755 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:46:10.936819 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:46:10.940057 (+ 3238us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:10.943434 (+ 3377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:10.943439 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:10.943442 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:10.943559 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:10.945193 (+ 1634us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:10.945198 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:10.945202 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:10.945550 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:46:10.945683 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0619 16:46:10.947967 (+ 2284us) server_negotiation.cc:299] Negotiation successful
0619 16:46:10.948292 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":7955,"thread_start_us":7817,"threads_started":1}
I20260619 16:46:10.951678 13732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:10.901606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43371 (local address 127.0.97.3:40583)
0619 16:46:10.913394 (+ 11788us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:10.913446 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0619 16:46:10.915996 (+ 2550us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:46:10.917890 (+ 1894us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:46:10.917904 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:46:10.918560 (+ 656us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:46:10.919393 (+ 833us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:10.919419 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:10.923519 (+ 4100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:10.923524 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:10.925617 (+ 2093us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:10.925630 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:10.926956 (+ 1326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:10.926963 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:10.929061 (+ 2098us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:10.929868 (+ 807us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:46:10.929905 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:46:10.936437 (+ 6532us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:46:10.940688 (+ 4251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:46:10.940698 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:46:10.940730 (+ 32us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:46:10.942479 (+ 1749us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:46:10.943700 (+ 1221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:46:10.943705 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:46:10.943725 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:46:10.943963 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:46:10.947093 (+ 3130us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:46:10.947101 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:46:10.947538 (+ 437us) client_negotiation.cc:769] Sending connection context
0619 16:46:10.949499 (+ 1961us) client_negotiation.cc:240] Negotiation successful
0619 16:46:10.950326 (+ 827us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11457,"thread_start_us":15067,"threads_started":1}
I20260619 16:46:10.953159 13730 heartbeater.cc:347] Connected to a master server at 127.0.97.62:43371
I20260619 16:46:10.953483 13730 heartbeater.cc:464] Registering TS with master...
I20260619 16:46:10.954368 13730 heartbeater.cc:511] Master 127.0.97.62:43371 requested a full tablet report, sending...
I20260619 16:46:10.955895 13254 ts_manager.cc:194] Registered new tserver with Master: 0e27e43e6d214571870edb0b771dbf91 (127.0.97.3:35299)
I20260619 16:46:10.957000 13254 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:40583
I20260619 16:46:10.962387 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 16:46:10.994653 13733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:10.969541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51430 (local address 127.0.97.62:43371)
0619 16:46:10.969785 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:10.969791 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:10.978329 (+ 8538us) server_negotiation.cc:407] Connection header received
0619 16:46:10.978508 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:10.978514 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:10.978598 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:10.978709 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:46:10.980672 (+ 1963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:10.981781 (+ 1109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:10.983658 (+ 1877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:10.984318 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:10.984470 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:10.988161 (+ 3691us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:46:10.988168 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:46:10.988172 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:46:10.988215 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:46:10.991672 (+ 3457us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:10.992515 (+ 843us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:10.992521 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:10.992525 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:10.992682 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:10.993093 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:10.993098 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:10.993101 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:10.993471 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:46:10.993720 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0619 16:46:10.994082 (+ 362us) server_negotiation.cc:299] Negotiation successful
0619 16:46:10.994325 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260619 16:46:11.002256 13254 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51430:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 16:46:11.013957 13254 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 16:46:11.068540 13755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:11.029471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42815 (local address 127.0.0.1:38052)
0619 16:46:11.053966 (+ 24495us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:11.054016 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0619 16:46:11.054253 (+ 237us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:46:11.056311 (+ 2058us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:46:11.056318 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:46:11.056356 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:46:11.056757 (+ 401us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:11.056770 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:11.060686 (+ 3916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:11.060691 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:11.064375 (+ 3684us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:11.064390 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:11.067598 (+ 3208us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:11.067604 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:11.067805 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:11.067863 (+ 58us) client_negotiation.cc:769] Sending connection context
0619 16:46:11.067986 (+ 123us) client_negotiation.cc:240] Negotiation successful
0619 16:46:11.068182 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23526,"thread_start_us":30203,"threads_started":1}
I20260619 16:46:11.069958 13754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:11.030135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38052 (local address 127.0.97.1:42815)
0619 16:46:11.053826 (+ 23691us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:11.053836 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:11.055483 (+ 1647us) server_negotiation.cc:407] Connection header received
0619 16:46:11.055596 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:11.055604 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:11.055857 (+ 253us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:11.056037 (+ 180us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:46:11.056940 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:11.060538 (+ 3598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:11.064556 (+ 4018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:11.067096 (+ 2540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:11.067223 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:11.067449 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0619 16:46:11.068903 (+ 1454us) server_negotiation.cc:299] Negotiation successful
0619 16:46:11.069158 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23492,"thread_start_us":29194,"threads_started":1}
I20260619 16:46:11.078410 13755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:11.064249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:44875 (local address 127.0.0.1:48574)
0619 16:46:11.069475 (+ 5226us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:11.069492 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 16:46:11.069590 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:46:11.070498 (+ 908us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:46:11.070503 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:46:11.070531 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:46:11.070730 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:11.070740 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:11.072741 (+ 2001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:11.072747 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:11.074695 (+ 1948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:11.074710 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:11.076878 (+ 2168us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:11.076885 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:11.077084 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:11.077843 (+ 759us) client_negotiation.cc:769] Sending connection context
0619 16:46:11.077935 (+ 92us) client_negotiation.cc:240] Negotiation successful
0619 16:46:11.078087 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5120,"threads_started":1}
I20260619 16:46:11.079638 13756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:11.060574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48574 (local address 127.0.97.2:44875)
0619 16:46:11.064224 (+ 3650us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:11.064232 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:11.069569 (+ 5337us) server_negotiation.cc:407] Connection header received
0619 16:46:11.069755 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:11.069765 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:11.070058 (+ 293us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:11.070265 (+ 207us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:46:11.070929 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:11.072567 (+ 1638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:11.074890 (+ 2323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:11.076412 (+ 1522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:11.076535 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:11.076784 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0619 16:46:11.079164 (+ 2380us) server_negotiation.cc:299] Negotiation successful
0619 16:46:11.079334 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3502,"thread_start_us":3340,"threads_started":1}
I20260619 16:46:11.082886 13530 tablet_service.cc:1467] Processing CreateTablet for tablet 6efedf4c7e924fd484f40bd78f2083c9 (DEFAULT_TABLE table=test-table [id=6a443809cf6a4780b92838d073593e0d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:46:11.082955 13386 tablet_service.cc:1467] Processing CreateTablet for tablet 6efedf4c7e924fd484f40bd78f2083c9 (DEFAULT_TABLE table=test-table [id=6a443809cf6a4780b92838d073593e0d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:46:11.084391 13530 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6efedf4c7e924fd484f40bd78f2083c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:46:11.084749 13386 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6efedf4c7e924fd484f40bd78f2083c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:46:11.090966 13752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:11.029539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:35299 (local address 127.0.0.1:57356)
0619 16:46:11.059925 (+ 30386us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:11.059942 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 16:46:11.060046 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:46:11.072404 (+ 12358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:46:11.072409 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:46:11.072434 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:46:11.072801 (+ 367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:11.072816 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:11.075127 (+ 2311us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:11.075133 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:11.078437 (+ 3304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:11.078451 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:11.089817 (+ 11366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:11.089835 (+ 18us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:11.090056 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:11.090099 (+ 43us) client_negotiation.cc:769] Sending connection context
0619 16:46:11.090186 (+ 87us) client_negotiation.cc:240] Negotiation successful
0619 16:46:11.090343 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30263,"thread_start_us":23761,"threads_started":1}
I20260619 16:46:11.093061 13753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:11.030123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57356 (local address 127.0.97.3:35299)
0619 16:46:11.071579 (+ 41456us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:11.071588 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:11.071613 (+ 25us) server_negotiation.cc:407] Connection header received
0619 16:46:11.071698 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:11.071706 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:11.071945 (+ 239us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:11.073005 (+ 1060us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:46:11.073308 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:11.074963 (+ 1655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:11.078607 (+ 3644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:11.080035 (+ 1428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:11.091570 (+ 11535us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:11.092471 (+ 901us) server_negotiation.cc:1035] Waiting for connection context
0619 16:46:11.092545 (+ 74us) server_negotiation.cc:299] Negotiation successful
0619 16:46:11.092726 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41293,"thread_start_us":46972,"threads_started":1}
I20260619 16:46:11.096910 13665 tablet_service.cc:1467] Processing CreateTablet for tablet 6efedf4c7e924fd484f40bd78f2083c9 (DEFAULT_TABLE table=test-table [id=6a443809cf6a4780b92838d073593e0d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:46:11.098519 13665 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6efedf4c7e924fd484f40bd78f2083c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:46:11.104489 13759 tablet_bootstrap.cc:492] T 6efedf4c7e924fd484f40bd78f2083c9 P d6b020ec64a941d6ad345b788fa9a341: Bootstrap starting.
I20260619 16:46:11.106912 13758 tablet_bootstrap.cc:492] T 6efedf4c7e924fd484f40bd78f2083c9 P d4e75f2cdfcf45a787e5007b8a111a2c: Bootstrap starting.
I20260619 16:46:11.110221 13759 tablet_bootstrap.cc:654] T 6efedf4c7e924fd484f40bd78f2083c9 P d6b020ec64a941d6ad345b788fa9a341: Neither blocks nor log segments found. Creating new log.
I20260619 16:46:11.111426 13759 log.cc:824] T 6efedf4c7e924fd484f40bd78f2083c9 P d6b020ec64a941d6ad345b788fa9a341: Log is configured to *not* fsync() on all Append() calls
I20260619 16:46:11.112313 13758 tablet_bootstrap.cc:654] T 6efedf4c7e924fd484f40bd78f2083c9 P d4e75f2cdfcf45a787e5007b8a111a2c: Neither blocks nor log segments found. Creating new log.
I20260619 16:46:11.113580 13758 log.cc:824] T 6efedf4c7e924fd484f40bd78f2083c9 P d4e75f2cdfcf45a787e5007b8a111a2c: Log is configured to *not* fsync() on all Append() calls
I20260619 16:46:11.118883 13761 tablet_bootstrap.cc:492] T 6efedf4c7e924fd484f40bd78f2083c9 P 0e27e43e6d214571870edb0b771dbf91: Bootstrap starting.
I20260619 16:46:11.124262 13761 tablet_bootstrap.cc:654] T 6efedf4c7e924fd484f40bd78f2083c9 P 0e27e43e6d214571870edb0b771dbf91: Neither blocks nor log segments found. Creating new log.
I20260619 16:46:11.125430 13761 log.cc:824] T 6efedf4c7e924fd484f40bd78f2083c9 P 0e27e43e6d214571870edb0b771dbf91: Log is configured to *not* fsync() on all Append() calls
I20260619 16:46:11.126031 13758 tablet_bootstrap.cc:492] T 6efedf4c7e924fd484f40bd78f2083c9 P d4e75f2cdfcf45a787e5007b8a111a2c: No bootstrap required, opened a new log
I20260619 16:46:11.126360 13758 ts_tablet_manager.cc:1397] T 6efedf4c7e924fd484f40bd78f2083c9 P d4e75f2cdfcf45a787e5007b8a111a2c: Time spent bootstrapping tablet: real 0.020s user 0.011s sys 0.005s
I20260619 16:46:11.127879 13759 tablet_bootstrap.cc:492] T 6efedf4c7e924fd484f40bd78f2083c9 P d6b020ec64a941d6ad345b788fa9a341: No bootstrap required, opened a new log
I20260619 16:46:11.128228 13759 ts_tablet_manager.cc:1397] T 6efedf4c7e924fd484f40bd78f2083c9 P d6b020ec64a941d6ad345b788fa9a341: Time spent bootstrapping tablet: real 0.024s user 0.005s sys 0.012s
I20260619 16:46:11.137621 13758 raft_consensus.cc:348] T 6efedf4c7e924fd484f40bd78f2083c9 P d4e75f2cdfcf45a787e5007b8a111a2c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b020ec64a941d6ad345b788fa9a341" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42815 } } peers { permanent_uuid: "0e27e43e6d214571870edb0b771dbf91" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35299 } } peers { permanent_uuid: "d4e75f2cdfcf45a787e5007b8a111a2c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44875 } }
I20260619 16:46:11.138283 13758 raft_consensus.cc:374] T 6efedf4c7e924fd484f40bd78f2083c9 P d4e75f2cdfcf45a787e5007b8a111a2c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:46:11.138409 13758 raft_consensus.cc:729] T 6efedf4c7e924fd484f40bd78f2083c9 P d4e75f2cdfcf45a787e5007b8a111a2c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4e75f2cdfcf45a787e5007b8a111a2c, State: Initialized, Role: FOLLOWER
I20260619 16:46:11.139031 13758 consensus_queue.cc:260] T 6efedf4c7e924fd484f40bd78f2083c9 P d4e75f2cdfcf45a787e5007b8a111a2c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b020ec64a941d6ad345b788fa9a341" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42815 } } peers { permanent_uuid: "0e27e43e6d214571870edb0b771dbf91" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35299 } } peers { permanent_uuid: "d4e75f2cdfcf45a787e5007b8a111a2c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44875 } }
I20260619 16:46:11.140938 13759 raft_consensus.cc:348] T 6efedf4c7e924fd484f40bd78f2083c9 P d6b020ec64a941d6ad345b788fa9a341 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b020ec64a941d6ad345b788fa9a341" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42815 } } peers { permanent_uuid: "0e27e43e6d214571870edb0b771dbf91" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35299 } } peers { permanent_uuid: "d4e75f2cdfcf45a787e5007b8a111a2c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44875 } }
I20260619 16:46:11.141407 13761 tablet_bootstrap.cc:492] T 6efedf4c7e924fd484f40bd78f2083c9 P 0e27e43e6d214571870edb0b771dbf91: No bootstrap required, opened a new log
I20260619 16:46:11.141489 13759 raft_consensus.cc:374] T 6efedf4c7e924fd484f40bd78f2083c9 P d6b020ec64a941d6ad345b788fa9a341 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:46:11.141623 13759 raft_consensus.cc:729] T 6efedf4c7e924fd484f40bd78f2083c9 P d6b020ec64a941d6ad345b788fa9a341 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6b020ec64a941d6ad345b788fa9a341, State: Initialized, Role: FOLLOWER
I20260619 16:46:11.141791 13761 ts_tablet_manager.cc:1397] T 6efedf4c7e924fd484f40bd78f2083c9 P 0e27e43e6d214571870edb0b771dbf91: Time spent bootstrapping tablet: real 0.023s user 0.008s sys 0.012s
I20260619 16:46:11.142351 13759 consensus_queue.cc:260] T 6efedf4c7e924fd484f40bd78f2083c9 P d6b020ec64a941d6ad345b788fa9a341 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b020ec64a941d6ad345b788fa9a341" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42815 } } peers { permanent_uuid: "0e27e43e6d214571870edb0b771dbf91" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35299 } } peers { permanent_uuid: "d4e75f2cdfcf45a787e5007b8a111a2c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44875 } }
I20260619 16:46:11.150391 13759 ts_tablet_manager.cc:1428] T 6efedf4c7e924fd484f40bd78f2083c9 P d6b020ec64a941d6ad345b788fa9a341: Time spent starting tablet: real 0.022s user 0.018s sys 0.000s
I20260619 16:46:11.150792 13451 heartbeater.cc:503] Master 127.0.97.62:43371 was elected leader, sending a full tablet report...
I20260619 16:46:11.152809 13761 raft_consensus.cc:348] T 6efedf4c7e924fd484f40bd78f2083c9 P 0e27e43e6d214571870edb0b771dbf91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b020ec64a941d6ad345b788fa9a341" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42815 } } peers { permanent_uuid: "0e27e43e6d214571870edb0b771dbf91" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35299 } } peers { permanent_uuid: "d4e75f2cdfcf45a787e5007b8a111a2c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44875 } }
I20260619 16:46:11.153403 13761 raft_consensus.cc:374] T 6efedf4c7e924fd484f40bd78f2083c9 P 0e27e43e6d214571870edb0b771dbf91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:46:11.153544 13761 raft_consensus.cc:729] T 6efedf4c7e924fd484f40bd78f2083c9 P 0e27e43e6d214571870edb0b771dbf91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e27e43e6d214571870edb0b771dbf91, State: Initialized, Role: FOLLOWER
I20260619 16:46:11.154356 13761 consensus_queue.cc:260] T 6efedf4c7e924fd484f40bd78f2083c9 P 0e27e43e6d214571870edb0b771dbf91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b020ec64a941d6ad345b788fa9a341" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42815 } } peers { permanent_uuid: "0e27e43e6d214571870edb0b771dbf91" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35299 } } peers { permanent_uuid: "d4e75f2cdfcf45a787e5007b8a111a2c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44875 } }
I20260619 16:46:11.155365 13758 ts_tablet_manager.cc:1428] T 6efedf4c7e924fd484f40bd78f2083c9 P d4e75f2cdfcf45a787e5007b8a111a2c: Time spent starting tablet: real 0.029s user 0.021s sys 0.007s
I20260619 16:46:11.156301 13730 heartbeater.cc:503] Master 127.0.97.62:43371 was elected leader, sending a full tablet report...
I20260619 16:46:11.156908 13761 ts_tablet_manager.cc:1428] T 6efedf4c7e924fd484f40bd78f2083c9 P 0e27e43e6d214571870edb0b771dbf91: Time spent starting tablet: real 0.015s user 0.011s sys 0.003s
W20260619 16:46:11.162879 13596 tablet.cc:2369] T 6efedf4c7e924fd484f40bd78f2083c9 P d4e75f2cdfcf45a787e5007b8a111a2c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 16:46:11.170378 13731 tablet.cc:2369] T 6efedf4c7e924fd484f40bd78f2083c9 P 0e27e43e6d214571870edb0b771dbf91: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 16:46:11.271971 13765 raft_consensus.cc:482] T 6efedf4c7e924fd484f40bd78f2083c9 P d6b020ec64a941d6ad345b788fa9a341 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 16:46:11.272287 13765 raft_consensus.cc:504] T 6efedf4c7e924fd484f40bd78f2083c9 P d6b020ec64a941d6ad345b788fa9a341 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b020ec64a941d6ad345b788fa9a341" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42815 } } peers { permanent_uuid: "0e27e43e6d214571870edb0b771dbf91" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35299 } } peers { permanent_uuid: "d4e75f2cdfcf45a787e5007b8a111a2c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44875 } }
I20260619 16:46:11.274170 13765 leader_election.cc:288] T 6efedf4c7e924fd484f40bd78f2083c9 P d6b020ec64a941d6ad345b788fa9a341 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0e27e43e6d214571870edb0b771dbf91 (127.0.97.3:35299), d4e75f2cdfcf45a787e5007b8a111a2c (127.0.97.2:44875)
I20260619 16:46:11.282788 13753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:11.275239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:42113 (local address 127.0.97.3:35299)
0619 16:46:11.275426 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:11.275432 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:11.276067 (+ 635us) server_negotiation.cc:407] Connection header received
0619 16:46:11.276342 (+ 275us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:11.276346 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:11.276416 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:11.276490 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:46:11.276950 (+ 460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:11.277864 (+ 914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:11.280430 (+ 2566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:11.281630 (+ 1200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:11.281720 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:11.281841 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0619 16:46:11.282396 (+ 555us) server_negotiation.cc:299] Negotiation successful
0619 16:46:11.282539 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260619 16:46:11.283097 13756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:11.274855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:56533 (local address 127.0.97.2:44875)
0619 16:46:11.275066 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:11.275073 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:11.276303 (+ 1230us) server_negotiation.cc:407] Connection header received
0619 16:46:11.276557 (+ 254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:11.276562 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:11.276642 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:11.276697 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:46:11.277394 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:11.278305 (+ 911us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:11.280810 (+ 2505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:11.281987 (+ 1177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:11.282086 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:11.282176 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0619 16:46:11.282707 (+ 531us) server_negotiation.cc:299] Negotiation successful
0619 16:46:11.282847 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260619 16:46:11.283946 13768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:11.274396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:35299 (local address 127.0.97.1:42113)
0619 16:46:11.275950 (+ 1554us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:11.275971 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 16:46:11.276190 (+ 219us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:46:11.276519 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:46:11.276524 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:46:11.276547 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:46:11.276811 (+ 264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:11.276819 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:11.277985 (+ 1166us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:11.277990 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:11.280251 (+ 2261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:11.280269 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:11.282030 (+ 1761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:11.282036 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:11.282194 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:11.282236 (+ 42us) client_negotiation.cc:769] Sending connection context
0619 16:46:11.283367 (+ 1131us) client_negotiation.cc:240] Negotiation successful
0619 16:46:11.283495 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1405,"mutex_wait_us":4,"thread_start_us":1557,"threads_started":1}
I20260619 16:46:11.284013 13767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:11.274556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:44875 (local address 127.0.97.1:56533)
0619 16:46:11.276199 (+ 1643us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:11.276223 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0619 16:46:11.276415 (+ 192us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:46:11.276882 (+ 467us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:46:11.276886 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:46:11.276907 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:46:11.277227 (+ 320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:11.277237 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:11.278490 (+ 1253us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:11.278495 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:11.280638 (+ 2143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:11.280652 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:11.282373 (+ 1721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:11.282379 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:11.282542 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:11.282578 (+ 36us) client_negotiation.cc:769] Sending connection context
0619 16:46:11.283296 (+ 718us) client_negotiation.cc:240] Negotiation successful
0619 16:46:11.283466 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1531,"mutex_wait_us":50,"thread_start_us":1030,"threads_started":1}
I20260619 16:46:11.285106 13685 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6efedf4c7e924fd484f40bd78f2083c9" candidate_uuid: "d6b020ec64a941d6ad345b788fa9a341" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e27e43e6d214571870edb0b771dbf91" is_pre_election: true
I20260619 16:46:11.285163 13550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6efedf4c7e924fd484f40bd78f2083c9" candidate_uuid: "d6b020ec64a941d6ad345b788fa9a341" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4e75f2cdfcf45a787e5007b8a111a2c" is_pre_election: true
I20260619 16:46:11.285645 13550 raft_consensus.cc:2454] T 6efedf4c7e924fd484f40bd78f2083c9 P d4e75f2cdfcf45a787e5007b8a111a2c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d6b020ec64a941d6ad345b788fa9a341 in term 0.
I20260619 16:46:11.285647 13685 raft_consensus.cc:2454] T 6efedf4c7e924fd484f40bd78f2083c9 P 0e27e43e6d214571870edb0b771dbf91 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d6b020ec64a941d6ad345b788fa9a341 in term 0.
I20260619 16:46:11.286362 13339 leader_election.cc:302] T 6efedf4c7e924fd484f40bd78f2083c9 P d6b020ec64a941d6ad345b788fa9a341 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0e27e43e6d214571870edb0b771dbf91, d6b020ec64a941d6ad345b788fa9a341; no voters:
I20260619 16:46:11.287006 13765 raft_consensus.cc:2781] T 6efedf4c7e924fd484f40bd78f2083c9 P d6b020ec64a941d6ad345b788fa9a341 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 16:46:11.287137 13765 raft_consensus.cc:482] T 6efedf4c7e924fd484f40bd78f2083c9 P d6b020ec64a941d6ad345b788fa9a341 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 16:46:11.287221 13765 raft_consensus.cc:3037] T 6efedf4c7e924fd484f40bd78f2083c9 P d6b020ec64a941d6ad345b788fa9a341 [term 0 FOLLOWER]: Advancing to term 1
W20260619 16:46:11.290019 13452 tablet.cc:2369] T 6efedf4c7e924fd484f40bd78f2083c9 P d6b020ec64a941d6ad345b788fa9a341: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 16:46:11.290427 13765 raft_consensus.cc:504] T 6efedf4c7e924fd484f40bd78f2083c9 P d6b020ec64a941d6ad345b788fa9a341 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b020ec64a941d6ad345b788fa9a341" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42815 } } peers { permanent_uuid: "0e27e43e6d214571870edb0b771dbf91" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35299 } } peers { permanent_uuid: "d4e75f2cdfcf45a787e5007b8a111a2c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44875 } }
I20260619 16:46:11.291549 13550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6efedf4c7e924fd484f40bd78f2083c9" candidate_uuid: "d6b020ec64a941d6ad345b788fa9a341" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4e75f2cdfcf45a787e5007b8a111a2c"
I20260619 16:46:11.291757 13550 raft_consensus.cc:3037] T 6efedf4c7e924fd484f40bd78f2083c9 P d4e75f2cdfcf45a787e5007b8a111a2c [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:46:11.291841 13685 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6efedf4c7e924fd484f40bd78f2083c9" candidate_uuid: "d6b020ec64a941d6ad345b788fa9a341" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e27e43e6d214571870edb0b771dbf91"
I20260619 16:46:11.292030 13685 raft_consensus.cc:3037] T 6efedf4c7e924fd484f40bd78f2083c9 P 0e27e43e6d214571870edb0b771dbf91 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:46:11.292735 13765 leader_election.cc:288] T 6efedf4c7e924fd484f40bd78f2083c9 P d6b020ec64a941d6ad345b788fa9a341 [CANDIDATE]: Term 1 election: Requested vote from peers 0e27e43e6d214571870edb0b771dbf91 (127.0.97.3:35299), d4e75f2cdfcf45a787e5007b8a111a2c (127.0.97.2:44875)
I20260619 16:46:11.294446 13685 raft_consensus.cc:2454] T 6efedf4c7e924fd484f40bd78f2083c9 P 0e27e43e6d214571870edb0b771dbf91 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d6b020ec64a941d6ad345b788fa9a341 in term 1.
I20260619 16:46:11.294744 13550 raft_consensus.cc:2454] T 6efedf4c7e924fd484f40bd78f2083c9 P d4e75f2cdfcf45a787e5007b8a111a2c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d6b020ec64a941d6ad345b788fa9a341 in term 1.
I20260619 16:46:11.294788 13339 leader_election.cc:302] T 6efedf4c7e924fd484f40bd78f2083c9 P d6b020ec64a941d6ad345b788fa9a341 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0e27e43e6d214571870edb0b771dbf91, d6b020ec64a941d6ad345b788fa9a341; no voters:
I20260619 16:46:11.295094 13765 raft_consensus.cc:2781] T 6efedf4c7e924fd484f40bd78f2083c9 P d6b020ec64a941d6ad345b788fa9a341 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:46:11.295542 13765 raft_consensus.cc:686] T 6efedf4c7e924fd484f40bd78f2083c9 P d6b020ec64a941d6ad345b788fa9a341 [term 1 LEADER]: Becoming Leader. State: Replica: d6b020ec64a941d6ad345b788fa9a341, State: Running, Role: LEADER
I20260619 16:46:11.296042 13765 consensus_queue.cc:237] T 6efedf4c7e924fd484f40bd78f2083c9 P d6b020ec64a941d6ad345b788fa9a341 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b020ec64a941d6ad345b788fa9a341" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42815 } } peers { permanent_uuid: "0e27e43e6d214571870edb0b771dbf91" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35299 } } peers { permanent_uuid: "d4e75f2cdfcf45a787e5007b8a111a2c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44875 } }
I20260619 16:46:11.300920 13253 catalog_manager.cc:5495] T 6efedf4c7e924fd484f40bd78f2083c9 P d6b020ec64a941d6ad345b788fa9a341 reported cstate change: term changed from 0 to 1, leader changed from <none> to d6b020ec64a941d6ad345b788fa9a341 (127.0.97.1). New cstate: current_term: 1 leader_uuid: "d6b020ec64a941d6ad345b788fa9a341" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b020ec64a941d6ad345b788fa9a341" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42815 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0e27e43e6d214571870edb0b771dbf91" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35299 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d4e75f2cdfcf45a787e5007b8a111a2c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44875 } health_report { overall_health: UNKNOWN } } }
I20260619 16:46:11.360634 13754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:11.355487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38060 (local address 127.0.97.1:42815)
0619 16:46:11.355648 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:11.355652 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:11.355668 (+ 16us) server_negotiation.cc:407] Connection header received
0619 16:46:11.355748 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:11.355752 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:11.355836 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:11.355962 (+ 126us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 16:46:11.356382 (+ 420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:11.357127 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:11.358551 (+ 1424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:11.358973 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:11.359098 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:11.359565 (+ 467us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 16:46:11.360088 (+ 523us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 16:46:11.360224 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0619 16:46:11.360310 (+ 86us) server_negotiation.cc:299] Negotiation successful
0619 16:46:11.360386 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260619 16:46:11.368462 13685 raft_consensus.cc:1261] T 6efedf4c7e924fd484f40bd78f2083c9 P 0e27e43e6d214571870edb0b771dbf91 [term 1 FOLLOWER]: Refusing update from remote peer d6b020ec64a941d6ad345b788fa9a341: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:46:11.368462 13550 raft_consensus.cc:1261] T 6efedf4c7e924fd484f40bd78f2083c9 P d4e75f2cdfcf45a787e5007b8a111a2c [term 1 FOLLOWER]: Refusing update from remote peer d6b020ec64a941d6ad345b788fa9a341: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:46:11.369427 13769 consensus_queue.cc:1035] T 6efedf4c7e924fd484f40bd78f2083c9 P d6b020ec64a941d6ad345b788fa9a341 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e27e43e6d214571870edb0b771dbf91" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:46:11.369803 13765 consensus_queue.cc:1035] T 6efedf4c7e924fd484f40bd78f2083c9 P d6b020ec64a941d6ad345b788fa9a341 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4e75f2cdfcf45a787e5007b8a111a2c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44875 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:46:11.381630 13772 mvcc.cc:205] Tried to move back new op lower bound from 7298611492315090944 to 7298611492032757760. Current Snapshot: MvccSnapshot[applied={T|T < 7298611492315090944}]
I20260619 16:46:11.383831 13773 mvcc.cc:205] Tried to move back new op lower bound from 7298611492315090944 to 7298611492032757760. Current Snapshot: MvccSnapshot[applied={T|T < 7298611492315090944}]
I20260619 16:46:11.396023 13253 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51430:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:46:11.396271 13253 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51430:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:46:11.400013 13253 catalog_manager.cc:5782] T 00000000000000000000000000000000 P beda6ee725e9415687ecdf160264704b: Sending DeleteTablet for 3 replicas of tablet 6efedf4c7e924fd484f40bd78f2083c9
I20260619 16:46:11.401062 13386 tablet_service.cc:1514] Processing DeleteTablet for tablet 6efedf4c7e924fd484f40bd78f2083c9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:46:11 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:38052
I20260619 16:46:11.401299 13665 tablet_service.cc:1514] Processing DeleteTablet for tablet 6efedf4c7e924fd484f40bd78f2083c9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:46:11 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:57356
I20260619 16:46:11.401649 13530 tablet_service.cc:1514] Processing DeleteTablet for tablet 6efedf4c7e924fd484f40bd78f2083c9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:46:11 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:48574
I20260619 16:46:11.406562 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu pid 13291
I20260619 16:46:11.418454 13782 tablet_replica.cc:330] T 6efedf4c7e924fd484f40bd78f2083c9 P d6b020ec64a941d6ad345b788fa9a341: stopping tablet replica
I20260619 16:46:11.419049 13782 raft_consensus.cc:2229] T 6efedf4c7e924fd484f40bd78f2083c9 P d6b020ec64a941d6ad345b788fa9a341 [term 1 LEADER]: Raft consensus shutting down.
I20260619 16:46:11.419878 13782 raft_consensus.cc:2258] T 6efedf4c7e924fd484f40bd78f2083c9 P d6b020ec64a941d6ad345b788fa9a341 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:46:11.420701 13784 tablet_replica.cc:330] T 6efedf4c7e924fd484f40bd78f2083c9 P d4e75f2cdfcf45a787e5007b8a111a2c: stopping tablet replica
I20260619 16:46:11.422714 13782 ts_tablet_manager.cc:1905] T 6efedf4c7e924fd484f40bd78f2083c9 P d6b020ec64a941d6ad345b788fa9a341: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:46:11.422941 13784 raft_consensus.cc:2229] T 6efedf4c7e924fd484f40bd78f2083c9 P d4e75f2cdfcf45a787e5007b8a111a2c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 16:46:11.424213 13784 raft_consensus.cc:2258] T 6efedf4c7e924fd484f40bd78f2083c9 P d4e75f2cdfcf45a787e5007b8a111a2c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:46:11.426405 13784 ts_tablet_manager.cc:1905] T 6efedf4c7e924fd484f40bd78f2083c9 P d4e75f2cdfcf45a787e5007b8a111a2c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:46:11.428382 13754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:11.411346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38064 (local address 127.0.97.1:42815)
0619 16:46:11.411512 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:11.411519 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:11.411641 (+ 122us) server_negotiation.cc:407] Connection header received
0619 16:46:11.411834 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:11.411840 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:11.411920 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:11.412032 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:46:11.413707 (+ 1675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:11.415020 (+ 1313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:11.416905 (+ 1885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:11.417635 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:11.417768 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:11.421058 (+ 3290us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:46:11.421069 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:46:11.421086 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:46:11.421126 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:46:11.425406 (+ 4280us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:11.426235 (+ 829us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:11.426243 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:11.426246 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:11.426411 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:11.426822 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:11.426827 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:11.426830 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:11.427259 (+ 429us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:46:11.427532 (+ 273us) server_negotiation.cc:1035] Waiting for connection context
0619 16:46:11.427840 (+ 308us) server_negotiation.cc:299] Negotiation successful
0619 16:46:11.428047 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260619 16:46:11.429186 13446 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:46:11.432351 13783 tablet_replica.cc:330] T 6efedf4c7e924fd484f40bd78f2083c9 P 0e27e43e6d214571870edb0b771dbf91: stopping tablet replica
I20260619 16:46:11.432780 13783 raft_consensus.cc:2229] T 6efedf4c7e924fd484f40bd78f2083c9 P 0e27e43e6d214571870edb0b771dbf91 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 16:46:11.433216 13783 raft_consensus.cc:2258] T 6efedf4c7e924fd484f40bd78f2083c9 P 0e27e43e6d214571870edb0b771dbf91 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:46:11.434739 13783 ts_tablet_manager.cc:1905] T 6efedf4c7e924fd484f40bd78f2083c9 P 0e27e43e6d214571870edb0b771dbf91: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:46:11.435146 13784 ts_tablet_manager.cc:1918] T 6efedf4c7e924fd484f40bd78f2083c9 P d4e75f2cdfcf45a787e5007b8a111a2c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 16:46:11.435305 13784 log.cc:1196] T 6efedf4c7e924fd484f40bd78f2083c9 P d4e75f2cdfcf45a787e5007b8a111a2c: Deleting WAL directory at /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887385903558-388-0/minicluster-data/ts-1/wal/wals/6efedf4c7e924fd484f40bd78f2083c9
I20260619 16:46:11.435801 13784 ts_tablet_manager.cc:1939] T 6efedf4c7e924fd484f40bd78f2083c9 P d4e75f2cdfcf45a787e5007b8a111a2c: Deleting consensus metadata
I20260619 16:46:11.437573 13238 catalog_manager.cc:4841] TS d4e75f2cdfcf45a787e5007b8a111a2c (127.0.97.2:44875): tablet 6efedf4c7e924fd484f40bd78f2083c9 (table test-table [id=6a443809cf6a4780b92838d073593e0d]) successfully deleted
I20260619 16:46:11.439289 13783 ts_tablet_manager.cc:1918] T 6efedf4c7e924fd484f40bd78f2083c9 P 0e27e43e6d214571870edb0b771dbf91: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 16:46:11.439417 13783 log.cc:1196] T 6efedf4c7e924fd484f40bd78f2083c9 P 0e27e43e6d214571870edb0b771dbf91: Deleting WAL directory at /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887385903558-388-0/minicluster-data/ts-2/wal/wals/6efedf4c7e924fd484f40bd78f2083c9
I20260619 16:46:11.439793 13783 ts_tablet_manager.cc:1939] T 6efedf4c7e924fd484f40bd78f2083c9 P 0e27e43e6d214571870edb0b771dbf91: Deleting consensus metadata
I20260619 16:46:11.441078 13239 catalog_manager.cc:4841] TS 0e27e43e6d214571870edb0b771dbf91 (127.0.97.3:35299): tablet 6efedf4c7e924fd484f40bd78f2083c9 (table test-table [id=6a443809cf6a4780b92838d073593e0d]) successfully deleted
I20260619 16:46:11.525413 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu with pid 13291
I20260619 16:46:11.538808 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu pid 13453
W20260619 16:46:11.538864 13238 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.97.1:42815 (error 108)
I20260619 16:46:11.540439 13752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:11.540033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42815 (local address 127.0.0.1:38066)
0619 16:46:11.540181 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:11.540288 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:42815: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260619 16:46:11.540668 13238 catalog_manager.cc:4584] TS d6b020ec64a941d6ad345b788fa9a341 (127.0.97.1:42815): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 6efedf4c7e924fd484f40bd78f2083c9: Network error: Client connection negotiation failed: client connection to 127.0.97.1:42815: connect: Connection refused (error 111)
I20260619 16:46:11.552013 13756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:11.539781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48576 (local address 127.0.97.2:44875)
0619 16:46:11.539943 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:11.539946 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:11.539963 (+ 17us) server_negotiation.cc:407] Connection header received
0619 16:46:11.540014 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:11.540017 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:11.540095 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:11.540191 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:46:11.541828 (+ 1637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:11.542536 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:11.543850 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:11.544237 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:11.544345 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:11.546857 (+ 2512us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:46:11.546870 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:46:11.546883 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:46:11.546915 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:46:11.549499 (+ 2584us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:11.550200 (+ 701us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:11.550207 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:11.550210 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:11.550358 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:11.550692 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:11.550699 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:11.550701 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:11.551096 (+ 395us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:46:11.551354 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0619 16:46:11.551563 (+ 209us) server_negotiation.cc:299] Negotiation successful
0619 16:46:11.551731 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260619 16:46:11.552697 13590 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:46:11.594339 13752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:11.593864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42815 (local address 127.0.0.1:38082)
0619 16:46:11.594098 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:11.594187 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:42815: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120}
I20260619 16:46:11.595278 13752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:11.594986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42815 (local address 127.0.0.1:38096)
0619 16:46:11.595099 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:11.595166 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:42815: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260619 16:46:11.640400 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu with pid 13453
I20260619 16:46:11.653244 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu pid 13599
I20260619 16:46:11.656076 13752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:11.655642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42815 (local address 127.0.0.1:38112)
0619 16:46:11.655841 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:11.655916 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:42815: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260619 16:46:11.657147 13752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:11.656734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42815 (local address 127.0.0.1:38116)
0619 16:46:11.656916 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:11.657001 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:42815: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260619 16:46:11.666013 13753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:11.653971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57362 (local address 127.0.97.3:35299)
0619 16:46:11.654127 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:11.654131 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:11.654149 (+ 18us) server_negotiation.cc:407] Connection header received
0619 16:46:11.654305 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:11.654309 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:11.654373 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:11.654456 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:46:11.655795 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:11.656630 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:11.658177 (+ 1547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:11.658571 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:11.658696 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:11.660950 (+ 2254us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:46:11.660963 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:46:11.660979 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:46:11.661017 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:46:11.663470 (+ 2453us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:11.664111 (+ 641us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:11.664118 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:11.664121 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:11.664255 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:11.664576 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:11.664579 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:11.664581 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:11.664931 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:46:11.665150 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0619 16:46:11.665436 (+ 286us) server_negotiation.cc:299] Negotiation successful
0619 16:46:11.665582 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260619 16:46:11.666616 13725 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:46:11.737910 13752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:11.737373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42815 (local address 127.0.0.1:38128)
0619 16:46:11.737572 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:11.737684 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:42815: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260619 16:46:11.738936 13752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:11.738581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42815 (local address 127.0.0.1:38130)
0619 16:46:11.738748 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:11.738814 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:42815: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
I20260619 16:46:11.752455 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu with pid 13599
I20260619 16:46:11.765158 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu pid 13221
I20260619 16:46:11.766295 13284 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:46:11.875547 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu with pid 13221
I20260619 16:46:11.902711 388 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260619 16:46:12.236984 388 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260619 16:46:12.237160 388 mini_ranger.cc:66] Stopping Ranger...
I20260619 16:46:12.572158 388 mini_ranger.cc:68] Stopped Ranger
2026-06-19 16:46:12.572 UTC [12273] LOG: received smart shutdown request
2026-06-19 16:46:12.574 UTC [12273] LOG: background worker "logical replication launcher" (PID 12282) exited with exit code 1
2026-06-19 16:46:12.575 UTC [12277] LOG: shutting down
2026-06-19 16:46:12.713 UTC [12273] LOG: database system is shut down
2026-06-19T16:46:12Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (48574 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887385903558-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 16:46:12 dist-test-slave-wvz7 krb5kdc[13798](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 16:46:12 dist-test-slave-wvz7 krb5kdc[13798](info): set up 2 sockets
Jun 19 16:46:12 dist-test-slave-wvz7 krb5kdc[13798](info): commencing operation
krb5kdc: starting...
W20260619 16:46:13.825887 388 mini_kdc.cc:121] Time spent starting KDC: real 0.978s user 0.000s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887385903558-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887385903558-388-0/krb5kdc/test-user.keytab.
Jun 19 16:46:13 dist-test-slave-wvz7 krb5kdc[13798](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887573, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T16:46:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T16:46:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887385903558-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887385903558-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260619 16:46:14.029664 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu
/tmp/dist-test-taskEs2EH8/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887385903558-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887385903558-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887385903558-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887385903558-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:41079
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887385903558-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39809
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:41079,127.0.97.61:38025,127.0.97.60:33823
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887385903558-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887385903558-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887385903558-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:46:14.333550 13811 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:46:14.333834 13811 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:46:14.333972 13811 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:46:14.340958 13811 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:46:14.341053 13811 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:46:14.341104 13811 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:46:14.341180 13811 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:46:14.341240 13811 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:46:14.350816 13811 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39809
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887385903558-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887385903558-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:41079,127.0.97.61:38025,127.0.97.60:33823
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887385903558-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:41079
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887385903558-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887385903558-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:46:14.352058 13811 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:46:14.353281 13811 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:46:14.363775 13816 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:46:14.363883 13817 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:46:14.363968 13819 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:46:14.364598 13811 server_base.cc:1034] running on GCE node
I20260619 16:46:14.365273 13811 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:46:14.366582 13811 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:46:14.367774 13811 hybrid_clock.cc:648] HybridClock initialized: now 1781887574367733 us; error 37 us; skew 500 ppm
I20260619 16:46:14.368078 13811 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:46:14 dist-test-slave-wvz7 krb5kdc[13798](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887574, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:46:14.370954 13811 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260619 16:46:14.372234 13811 webserver.cc:460] Webserver started at http://127.0.97.62:40865/ using document root <none> and password file <none>
I20260619 16:46:14.372869 13811 fs_manager.cc:359] Metadata directory not provided
I20260619 16:46:14.372979 13811 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:46:14.373245 13811 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:46:14.376014 13811 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887385903558-388-0/minicluster-data/master-0/data/instance:
uuid: "6dd438a401754d518fa33f2f1b349330"
format_stamp: "Formatted at 2026-06-19 16:46:14 on dist-test-slave-wvz7"
server_key: "ece9a25c800071e7f105b923276b1b07"
server_key_iv: "21f1c01c69988dfa86b5f839d37fcf78"
server_key_version: "encryptionkey@0"
I20260619 16:46:14.376753 13811 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887385903558-388-0/minicluster-data/master-0/wal/instance:
uuid: "6dd438a401754d518fa33f2f1b349330"
format_stamp: "Formatted at 2026-06-19 16:46:14 on dist-test-slave-wvz7"
server_key: "ece9a25c800071e7f105b923276b1b07"
server_key_iv: "21f1c01c69988dfa86b5f839d37fcf78"
server_key_version: "encryptionkey@0"
I20260619 16:46:14.381475 13811 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.001s
I20260619 16:46:14.384670 13826 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:46:14.385960 13811 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 16:46:14.386107 13811 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887385903558-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887385903558-388-0/minicluster-data/master-0/wal
uuid: "6dd438a401754d518fa33f2f1b349330"
format_stamp: "Formatted at 2026-06-19 16:46:14 on dist-test-slave-wvz7"
server_key: "ece9a25c800071e7f105b923276b1b07"
server_key_iv: "21f1c01c69988dfa86b5f839d37fcf78"
server_key_version: "encryptionkey@0"
I20260619 16:46:14.386281 13811 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887385903558-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887385903558-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887385903558-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:46:14.397043 13811 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:46:14.400461 13811 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:46:14.400696 13811 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:46:14.416874 13811 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:41079
I20260619 16:46:14.416913 13877 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:41079 every 8 connection(s)
I20260619 16:46:14.418148 13811 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887385903558-388-0/minicluster-data/master-0/data/info.pb
I20260619 16:46:14.421458 13878 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:46:14.427932 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu as pid 13811
I20260619 16:46:14.427274 13878 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41079 } has no permanent_uuid. Determining permanent_uuid...
I20260619 16:46:14.428158 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887385903558-388-0/minicluster-data/master-0/wal/instance
I20260619 16:46:14.428951 388 external_mini_cluster.cc:1428] Setting key c6c38876aa2a5bcddb2f93090d41312d
Jun 19 16:46:14 dist-test-slave-wvz7 krb5kdc[13798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887574, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.61@KRBTEST.COM; defaulting to no policy
I20260619 16:46:14.448575 13880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:14.430631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56896 (local address 127.0.97.62:41079)
0619 16:46:14.431367 (+ 736us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:14.431384 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:14.431791 (+ 407us) server_negotiation.cc:407] Connection header received
0619 16:46:14.433652 (+ 1861us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:14.433657 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:14.433911 (+ 254us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:14.434076 (+ 165us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:46:14.435233 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:14.436413 (+ 1180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:14.437771 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:14.438294 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:14.438468 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:14.441913 (+ 3445us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:46:14.441922 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:46:14.441932 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:46:14.441961 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:46:14.444580 (+ 2619us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:14.445406 (+ 826us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:14.445412 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:14.445414 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:14.445555 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:14.446015 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:14.446020 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:14.446022 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:14.446552 (+ 530us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:46:14.446714 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0619 16:46:14.447324 (+ 610us) server_negotiation.cc:299] Negotiation successful
0619 16:46:14.447670 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":160,"server-negotiator.queue_time_us":518,"thread_start_us":341,"threads_started":1}
I20260619 16:46:14.448623 13881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:14.430631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41079 (local address 127.0.0.1:56896)
0619 16:46:14.431644 (+ 1013us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:14.431680 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0619 16:46:14.433333 (+ 1653us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:46:14.434112 (+ 779us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:46:14.434118 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:46:14.434752 (+ 634us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:46:14.435063 (+ 311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:14.435078 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:14.436546 (+ 1468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:14.436549 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:14.437635 (+ 1086us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:14.437651 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:14.438513 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:14.438520 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:14.438743 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:14.439303 (+ 560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:46:14.439325 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:46:14.441638 (+ 2313us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:46:14.444711 (+ 3073us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:46:14.444723 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:46:14.444735 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:46:14.445262 (+ 527us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:46:14.445682 (+ 420us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:46:14.445687 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:46:14.445691 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:46:14.445907 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:46:14.446716 (+ 809us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:46:14.446724 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:46:14.446989 (+ 265us) client_negotiation.cc:769] Sending connection context
0619 16:46:14.447183 (+ 194us) client_negotiation.cc:240] Negotiation successful
0619 16:46:14.447559 (+ 376us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":822,"mutex_wait_us":2,"spinlock_wait_cycles":16128,"thread_start_us":639,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.61@KRBTEST.COM" created.
I20260619 16:46:14.451385 13878 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38025 } has no permanent_uuid. Determining permanent_uuid...
I20260619 16:46:14.452313 13881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:14.451982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:38025 (local address 127.0.0.1:41282)
0619 16:46:14.452131 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:14.452220 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:38025: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260619 16:46:14.452548 13829 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.61:38025: connect: Connection refused (error 111)
I20260619 16:46:14.454006 13881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:14.453667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:38025 (local address 127.0.0.1:41292)
0619 16:46:14.453831 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:14.453896 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:38025: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
W20260619 16:46:14.454195 13878 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:38025: Network error: Client connection negotiation failed: client connection to 127.0.97.61:38025: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887385903558-388-0/krb5kdc/kudu_127.0.97.61.keytab.
Entry for principal kudu/127.0.97.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887385903558-388-0/krb5kdc/kudu_127.0.97.61.keytab.
I20260619 16:46:14.472121 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu
/tmp/dist-test-taskEs2EH8/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887385903558-388-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887385903558-388-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887385903558-388-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887385903558-388-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:38025
--webserver_interface=127.0.97.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887385903558-388-0/krb5kdc/kudu_127.0.97.61.keytab
--principal=kudu/127.0.97.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39809
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:41079,127.0.97.61:38025,127.0.97.60:33823
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887385903558-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887385903558-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887385903558-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 16:46:14.503490 13878 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38025 } attempt: 1
I20260619 16:46:14.504657 13881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:14.504152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:38025 (local address 127.0.0.1:41306)
0619 16:46:14.504374 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:14.504499 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:38025: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260619 16:46:14.505618 13881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:14.505251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:38025 (local address 127.0.0.1:41322)
0619 16:46:14.505380 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:14.505463 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:38025: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260619 16:46:14.505818 13878 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:38025: Network error: Client connection negotiation failed: client connection to 127.0.97.61:38025: connect: Connection refused (error 111)
I20260619 16:46:14.574074 13878 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38025 } attempt: 2
I20260619 16:46:14.575291 13881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:14.574730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:38025 (local address 127.0.0.1:41336)
0619 16:46:14.575012 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:14.575129 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:38025: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260619 16:46:14.576272 13881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:14.575899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:38025 (local address 127.0.0.1:41340)
0619 16:46:14.576044 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:14.576123 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:38025: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260619 16:46:14.576454 13878 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:38025: Network error: Client connection negotiation failed: client connection to 127.0.97.61:38025: connect: Connection refused (error 111)
I20260619 16:46:14.667722 13878 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38025 } attempt: 3
I20260619 16:46:14.668999 13881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:14.668446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:38025 (local address 127.0.0.1:41350)
0619 16:46:14.668677 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:14.668796 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:38025: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260619 16:46:14.670117 13881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:14.669721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:38025 (local address 127.0.0.1:41358)
0619 16:46:14.669888 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:14.669965 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:38025: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
W20260619 16:46:14.670315 13878 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:38025: Network error: Client connection negotiation failed: client connection to 127.0.97.61:38025: connect: Connection refused (error 111)
W20260619 16:46:14.775108 13885 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:46:14.775383 13885 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:46:14.775475 13885 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:46:14.782446 13885 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:46:14.782543 13885 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:46:14.782595 13885 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:46:14.782647 13885 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:46:14.782688 13885 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:46:14.792279 13885 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39809
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887385903558-388-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887385903558-388-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:41079,127.0.97.61:38025,127.0.97.60:33823
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887385903558-388-0/krb5kdc/kudu_127.0.97.61.keytab
--principal=kudu/127.0.97.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:38025
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887385903558-388-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.97.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887385903558-388-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:46:14.793484 13885 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:46:14.794731 13885 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:46:14.805817 13890 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:46:14.805689 13893 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:46:14.805784 13891 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:46:14.806571 13885 server_base.cc:1034] running on GCE node
I20260619 16:46:14.807256 13885 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:46:14.808550 13885 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:46:14.809759 13885 hybrid_clock.cc:648] HybridClock initialized: now 1781887574809673 us; error 62 us; skew 500 ppm
I20260619 16:46:14.810084 13885 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 16:46:14.813537 13878 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38025 } attempt: 4
I20260619 16:46:14.814617 13881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:14.814132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:38025 (local address 127.0.0.1:41374)
0619 16:46:14.814398 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:14.814490 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:38025: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134}
I20260619 16:46:14.815578 13881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:14.815257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:38025 (local address 127.0.0.1:41382)
0619 16:46:14.815372 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:14.815460 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:38025: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260619 16:46:14.815752 13878 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:38025: Network error: Client connection negotiation failed: client connection to 127.0.97.61:38025: connect: Connection refused (error 111)
Jun 19 16:46:14 dist-test-slave-wvz7 krb5kdc[13798](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887574, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:46:14.817015 13885 init.cc:377] Logged in from keytab as kudu/127.0.97.61@KRBTEST.COM (short username kudu)
I20260619 16:46:14.818522 13885 webserver.cc:460] Webserver started at http://127.0.97.61:42201/ using document root <none> and password file <none>
I20260619 16:46:14.819198 13885 fs_manager.cc:359] Metadata directory not provided
I20260619 16:46:14.819311 13885 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:46:14.819584 13885 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:46:14.822783 13885 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887385903558-388-0/minicluster-data/master-1/data/instance:
uuid: "f4593c435c314505ac60b4a5aa3eebe3"
format_stamp: "Formatted at 2026-06-19 16:46:14 on dist-test-slave-wvz7"
server_key: "1fccd97f43fc7570a230b9f0062f80bc"
server_key_iv: "6c8a772ef9806fc81fa80b13782cd9bc"
server_key_version: "encryptionkey@0"
I20260619 16:46:14.823777 13885 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887385903558-388-0/minicluster-data/master-1/wal/instance:
uuid: "f4593c435c314505ac60b4a5aa3eebe3"
format_stamp: "Formatted at 2026-06-19 16:46:14 on dist-test-slave-wvz7"
server_key: "1fccd97f43fc7570a230b9f0062f80bc"
server_key_iv: "6c8a772ef9806fc81fa80b13782cd9bc"
server_key_version: "encryptionkey@0"
I20260619 16:46:14.830987 13885 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.005s sys 0.000s
I20260619 16:46:14.835497 13900 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:46:14.837348 13885 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260619 16:46:14.837500 13885 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887385903558-388-0/minicluster-data/master-1/data,/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887385903558-388-0/minicluster-data/master-1/wal
uuid: "f4593c435c314505ac60b4a5aa3eebe3"
format_stamp: "Formatted at 2026-06-19 16:46:14 on dist-test-slave-wvz7"
server_key: "1fccd97f43fc7570a230b9f0062f80bc"
server_key_iv: "6c8a772ef9806fc81fa80b13782cd9bc"
server_key_version: "encryptionkey@0"
I20260619 16:46:14.837724 13885 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887385903558-388-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887385903558-388-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887385903558-388-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:46:14.863874 13885 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:46:14.868479 13885 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:46:14.868734 13885 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:46:14.885296 13885 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.61:38025
I20260619 16:46:14.885372 13951 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.61:38025 every 8 connection(s)
I20260619 16:46:14.886865 13885 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887385903558-388-0/minicluster-data/master-1/data/info.pb
I20260619 16:46:14.890442 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu as pid 13885
I20260619 16:46:14.890600 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887385903558-388-0/minicluster-data/master-1/wal/instance
I20260619 16:46:14.890920 13952 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:46:14.891044 388 external_mini_cluster.cc:1428] Setting key 35e6f35569d65f5a881a93da2c05aa96
I20260619 16:46:14.897212 13952 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41079 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.97.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.60@KRBTEST.COM" created.
Jun 19 16:46:14 dist-test-slave-wvz7 krb5kdc[13798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887574, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 16:46:14.917320 13880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:14.899822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56900 (local address 127.0.97.62:41079)
0619 16:46:14.900032 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:14.900036 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:14.901313 (+ 1277us) server_negotiation.cc:407] Connection header received
0619 16:46:14.903148 (+ 1835us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:14.903154 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:14.903235 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:14.903320 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:46:14.904786 (+ 1466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:14.905500 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:14.907047 (+ 1547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:14.907446 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:14.907558 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:14.911311 (+ 3753us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:46:14.911319 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:46:14.911324 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:46:14.911367 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:46:14.914523 (+ 3156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:14.915217 (+ 694us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:14.915223 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:14.915227 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:14.915367 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:14.915744 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:14.915748 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:14.915751 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:14.916085 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:46:14.916217 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0619 16:46:14.916917 (+ 700us) server_negotiation.cc:299] Negotiation successful
0619 16:46:14.917098 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260619 16:46:14.918658 13955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:14.900387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41079 (local address 127.0.0.1:56900)
0619 16:46:14.901157 (+ 770us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:14.901199 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0619 16:46:14.902883 (+ 1684us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:46:14.903557 (+ 674us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:46:14.903565 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:46:14.904081 (+ 516us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:46:14.904623 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:14.904641 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:14.905646 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:14.905649 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:14.906902 (+ 1253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:14.906914 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:14.907655 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:14.907662 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:14.907893 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:14.908483 (+ 590us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:46:14.908505 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:46:14.911067 (+ 2562us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:46:14.914673 (+ 3606us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:46:14.914680 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:46:14.914693 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:46:14.915100 (+ 407us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:46:14.915467 (+ 367us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:46:14.915470 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:46:14.915472 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:46:14.915651 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:46:14.916191 (+ 540us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:46:14.916197 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:46:14.916542 (+ 345us) client_negotiation.cc:769] Sending connection context
0619 16:46:14.916775 (+ 233us) client_negotiation.cc:240] Negotiation successful
0619 16:46:14.917270 (+ 495us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":532,"thread_start_us":323,"threads_started":1}
I20260619 16:46:14.920315 13952 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38025 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 16:46:14 dist-test-slave-wvz7 krb5kdc[13798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887574, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260619 16:46:14.933440 13955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:14.921017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:38025 (local address 127.0.0.1:41390)
0619 16:46:14.921782 (+ 765us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:14.921803 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 16:46:14.921904 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:46:14.922331 (+ 427us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:46:14.922335 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:46:14.922731 (+ 396us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:46:14.922862 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:14.922870 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:14.924147 (+ 1277us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:14.924153 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:14.924957 (+ 804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:14.924967 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:14.925698 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:14.925704 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:14.925924 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:14.926424 (+ 500us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:46:14.926449 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:46:14.928274 (+ 1825us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:46:14.931192 (+ 2918us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:46:14.931198 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:46:14.931202 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:46:14.931551 (+ 349us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:46:14.931868 (+ 317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:46:14.931872 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:46:14.931876 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:46:14.931999 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:46:14.932543 (+ 544us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:46:14.932547 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:46:14.932679 (+ 132us) client_negotiation.cc:769] Sending connection context
0619 16:46:14.932796 (+ 117us) client_negotiation.cc:240] Negotiation successful
0619 16:46:14.933051 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":649}
I20260619 16:46:14.933516 13957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:14.921257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41390 (local address 127.0.97.61:38025)
0619 16:46:14.921831 (+ 574us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:14.921839 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:14.921872 (+ 33us) server_negotiation.cc:407] Connection header received
0619 16:46:14.922026 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:14.922032 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:14.922203 (+ 171us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:14.922331 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:46:14.922999 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:14.924012 (+ 1013us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:14.925074 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:14.925575 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:14.925680 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:14.928454 (+ 2774us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:46:14.928464 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:46:14.928476 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:46:14.928503 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:46:14.931087 (+ 2584us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:14.931660 (+ 573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:14.931668 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:14.931670 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:14.931773 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:14.932097 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:14.932102 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:14.932106 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:14.932428 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:46:14.932616 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0619 16:46:14.932945 (+ 329us) server_negotiation.cc:299] Negotiation successful
0619 16:46:14.933159 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":378,"thread_start_us":246,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887385903558-388-0/krb5kdc/kudu_127.0.97.60.keytab.
Entry for principal kudu/127.0.97.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887385903558-388-0/krb5kdc/kudu_127.0.97.60.keytab.
I20260619 16:46:14.935207 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu
/tmp/dist-test-taskEs2EH8/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887385903558-388-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887385903558-388-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887385903558-388-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887385903558-388-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:33823
--webserver_interface=127.0.97.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887385903558-388-0/krb5kdc/kudu_127.0.97.60.keytab
--principal=kudu/127.0.97.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39809
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:41079,127.0.97.61:38025,127.0.97.60:33823
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887385903558-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887385903558-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887385903558-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 16:46:14.935928 13952 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33823 } has no permanent_uuid. Determining permanent_uuid...
I20260619 16:46:14.936992 13955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:14.936562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:33823 (local address 127.0.0.1:40276)
0619 16:46:14.936721 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:14.936839 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33823: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260619 16:46:14.937247 13904 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33823: connect: Connection refused (error 111)
I20260619 16:46:14.938813 13955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:14.938482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:33823 (local address 127.0.0.1:40286)
0619 16:46:14.938636 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:14.938712 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33823: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260619 16:46:14.939029 13952 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:33823: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33823: connect: Connection refused (error 111)
I20260619 16:46:14.988317 13952 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33823 } attempt: 1
I20260619 16:46:14.989464 13955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:14.988991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:33823 (local address 127.0.0.1:40300)
0619 16:46:14.989199 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:14.989309 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33823: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260619 16:46:14.990633 13955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:14.990211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:33823 (local address 127.0.0.1:40304)
0619 16:46:14.990430 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:14.990513 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33823: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
W20260619 16:46:14.990872 13952 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:33823: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33823: connect: Connection refused (error 111)
I20260619 16:46:15.059106 13952 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33823 } attempt: 2
I20260619 16:46:15.060231 13955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:15.059790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:33823 (local address 127.0.0.1:40310)
0619 16:46:15.059992 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:15.060088 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33823: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260619 16:46:15.061156 13955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:15.060828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:33823 (local address 127.0.0.1:40324)
0619 16:46:15.060982 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:15.061047 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33823: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260619 16:46:15.061347 13952 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:33823: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33823: connect: Connection refused (error 111)
I20260619 16:46:15.115001 13878 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38025 } attempt: 5
Jun 19 16:46:15 dist-test-slave-wvz7 krb5kdc[13798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887574, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260619 16:46:15.129038 13881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:15.115753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:38025 (local address 127.0.0.1:41404)
0619 16:46:15.115969 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:15.115991 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0619 16:46:15.116105 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:46:15.116487 (+ 382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:46:15.116490 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:46:15.116854 (+ 364us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:46:15.117023 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:15.117032 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:15.118484 (+ 1452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:15.118489 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:15.119338 (+ 849us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:15.119348 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:15.120295 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:15.120300 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:15.120468 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:15.121080 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:46:15.121098 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:46:15.123268 (+ 2170us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:46:15.126848 (+ 3580us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:46:15.126851 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:46:15.126854 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:46:15.127208 (+ 354us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:46:15.127563 (+ 355us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:46:15.127566 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:46:15.127568 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:46:15.127678 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:46:15.128257 (+ 579us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:46:15.128261 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:46:15.128419 (+ 158us) client_negotiation.cc:769] Sending connection context
0619 16:46:15.128551 (+ 132us) client_negotiation.cc:240] Negotiation successful
0619 16:46:15.128759 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":73}
I20260619 16:46:15.129176 13957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:15.115916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41404 (local address 127.0.97.61:38025)
0619 16:46:15.116137 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:15.116144 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:15.116168 (+ 24us) server_negotiation.cc:407] Connection header received
0619 16:46:15.116240 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:15.116247 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:15.116351 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:15.116475 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:46:15.117179 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:15.118310 (+ 1131us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:15.119497 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:15.120144 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:15.120267 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:15.123461 (+ 3194us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:46:15.123468 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:46:15.123473 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:46:15.123511 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:46:15.126710 (+ 3199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:15.127311 (+ 601us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:15.127317 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:15.127320 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:15.127475 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:15.127762 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:15.127767 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:15.127771 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:15.128117 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:46:15.128268 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0619 16:46:15.128667 (+ 399us) server_negotiation.cc:299] Negotiation successful
0619 16:46:15.128888 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260619 16:46:15.130261 13878 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33823 } has no permanent_uuid. Determining permanent_uuid...
I20260619 16:46:15.131230 13881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:15.130872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:33823 (local address 127.0.0.1:40338)
0619 16:46:15.131029 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:15.131098 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33823: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260619 16:46:15.132099 13881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:15.131801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:33823 (local address 127.0.0.1:40348)
0619 16:46:15.131931 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:15.132010 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33823: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260619 16:46:15.132273 13878 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:33823: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33823: connect: Connection refused (error 111)
I20260619 16:46:15.152542 13952 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33823 } attempt: 3
I20260619 16:46:15.153626 13955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:15.153184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:33823 (local address 127.0.0.1:40358)
0619 16:46:15.153382 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:15.153475 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33823: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260619 16:46:15.154650 13955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:15.154257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:33823 (local address 127.0.0.1:40370)
0619 16:46:15.154467 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:15.154539 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33823: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117}
W20260619 16:46:15.154836 13952 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:33823: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33823: connect: Connection refused (error 111)
I20260619 16:46:15.183488 13878 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33823 } attempt: 1
I20260619 16:46:15.184530 13881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:15.184091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:33823 (local address 127.0.0.1:40372)
0619 16:46:15.184282 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:15.184370 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33823: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260619 16:46:15.185484 13881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:15.185160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:33823 (local address 127.0.0.1:40386)
0619 16:46:15.185301 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:15.185373 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33823: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260619 16:46:15.185673 13878 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:33823: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33823: connect: Connection refused (error 111)
W20260619 16:46:15.245536 13959 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:46:15.245854 13959 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:46:15.246006 13959 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:46:15.253013 13959 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:46:15.253122 13959 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:46:15.253198 13959 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:46:15.253273 13959 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:46:15.253334 13959 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:46:15.253880 13878 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33823 } attempt: 2
I20260619 16:46:15.254892 13881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:15.254462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:33823 (local address 127.0.0.1:40392)
0619 16:46:15.254657 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:15.254736 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33823: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260619 16:46:15.255853 13881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:15.255543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:33823 (local address 127.0.0.1:40400)
0619 16:46:15.255682 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:15.255750 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33823: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260619 16:46:15.256055 13878 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:33823: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33823: connect: Connection refused (error 111)
I20260619 16:46:15.263257 13959 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39809
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887385903558-388-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887385903558-388-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:41079,127.0.97.61:38025,127.0.97.60:33823
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887385903558-388-0/krb5kdc/kudu_127.0.97.60.keytab
--principal=kudu/127.0.97.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:33823
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887385903558-388-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.97.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887385903558-388-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:46:15.264487 13959 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:46:15.265690 13959 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:46:15.276561 13965 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:46:15.276643 13964 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:46:15.276592 13967 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:46:15.277331 13959 server_base.cc:1034] running on GCE node
I20260619 16:46:15.278121 13959 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:46:15.279377 13959 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:46:15.280586 13959 hybrid_clock.cc:648] HybridClock initialized: now 1781887575280536 us; error 39 us; skew 500 ppm
I20260619 16:46:15.280900 13959 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:46:15 dist-test-slave-wvz7 krb5kdc[13798](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887575, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:46:15.283809 13959 init.cc:377] Logged in from keytab as kudu/127.0.97.60@KRBTEST.COM (short username kudu)
I20260619 16:46:15.285151 13959 webserver.cc:460] Webserver started at http://127.0.97.60:44335/ using document root <none> and password file <none>
I20260619 16:46:15.285836 13959 fs_manager.cc:359] Metadata directory not provided
I20260619 16:46:15.285938 13959 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:46:15.286178 13959 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:46:15.288892 13959 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887385903558-388-0/minicluster-data/master-2/data/instance:
uuid: "34bdd0b575de4d37b6d6c5760c7bc982"
format_stamp: "Formatted at 2026-06-19 16:46:15 on dist-test-slave-wvz7"
server_key: "32d1ae3fb98cef8a827c491f26e30b1f"
server_key_iv: "2bb1b3336dc665aa97608ddb3921eb85"
server_key_version: "encryptionkey@0"
I20260619 16:46:15.289635 13959 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887385903558-388-0/minicluster-data/master-2/wal/instance:
uuid: "34bdd0b575de4d37b6d6c5760c7bc982"
format_stamp: "Formatted at 2026-06-19 16:46:15 on dist-test-slave-wvz7"
server_key: "32d1ae3fb98cef8a827c491f26e30b1f"
server_key_iv: "2bb1b3336dc665aa97608ddb3921eb85"
server_key_version: "encryptionkey@0"
I20260619 16:46:15.294486 13959 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260619 16:46:15.297782 13974 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:46:15.298023 13952 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33823 } attempt: 4
I20260619 16:46:15.299085 13955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:15.298576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:33823 (local address 127.0.0.1:40410)
0619 16:46:15.298795 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:15.298896 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33823: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260619 16:46:15.299342 13959 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260619 16:46:15.299515 13959 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887385903558-388-0/minicluster-data/master-2/data,/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887385903558-388-0/minicluster-data/master-2/wal
uuid: "34bdd0b575de4d37b6d6c5760c7bc982"
format_stamp: "Formatted at 2026-06-19 16:46:15 on dist-test-slave-wvz7"
server_key: "32d1ae3fb98cef8a827c491f26e30b1f"
server_key_iv: "2bb1b3336dc665aa97608ddb3921eb85"
server_key_version: "encryptionkey@0"
I20260619 16:46:15.299717 13959 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887385903558-388-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887385903558-388-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887385903558-388-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:46:15.300074 13955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:15.299712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:33823 (local address 127.0.0.1:40426)
0619 16:46:15.299883 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:15.299953 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33823: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97}
W20260619 16:46:15.300257 13952 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:33823: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33823: connect: Connection refused (error 111)
I20260619 16:46:15.331516 13959 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:46:15.335728 13959 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:46:15.335964 13959 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:46:15.351956 13959 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.60:33823
I20260619 16:46:15.351989 14025 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.60:33823 every 8 connection(s)
I20260619 16:46:15.353188 13959 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887385903558-388-0/minicluster-data/master-2/data/info.pb
I20260619 16:46:15.355020 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu as pid 13959
I20260619 16:46:15.355183 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887385903558-388-0/minicluster-data/master-2/wal/instance
I20260619 16:46:15.355638 388 external_mini_cluster.cc:1428] Setting key 18fb841593a6c5a0a85663350cc92135
I20260619 16:46:15.357064 14026 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:46:15.362228 13878 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33823 } attempt: 3
I20260619 16:46:15.366343 14026 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41079 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 16:46:15 dist-test-slave-wvz7 krb5kdc[13798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887573, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
Jun 19 16:46:15 dist-test-slave-wvz7 krb5kdc[13798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887574, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
Jun 19 16:46:15 dist-test-slave-wvz7 krb5kdc[13798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887575, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 16:46:15.378463 13881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:15.362759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:33823 (local address 127.0.0.1:40438)
0619 16:46:15.363031 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:15.363051 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 16:46:15.363154 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:46:15.363984 (+ 830us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:46:15.363990 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:46:15.364413 (+ 423us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:46:15.364590 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:15.364603 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:15.366055 (+ 1452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:15.366061 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:15.367226 (+ 1165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:15.367241 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:15.367967 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:15.367981 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:15.368150 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:15.368636 (+ 486us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:46:15.368659 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:46:15.371316 (+ 2657us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:46:15.375368 (+ 4052us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:46:15.375373 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:46:15.375378 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:46:15.375841 (+ 463us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:46:15.376337 (+ 496us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:46:15.376342 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:46:15.376345 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:46:15.376502 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:46:15.377410 (+ 908us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:46:15.377415 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:46:15.377609 (+ 194us) client_negotiation.cc:769] Sending connection context
0619 16:46:15.377811 (+ 202us) client_negotiation.cc:240] Negotiation successful
0619 16:46:15.378075 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138}
I20260619 16:46:15.379989 14030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:15.362948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40438 (local address 127.0.97.60:33823)
0619 16:46:15.363712 (+ 764us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:15.363716 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:15.363734 (+ 18us) server_negotiation.cc:407] Connection header received
0619 16:46:15.363769 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:15.363772 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:15.363820 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:15.363914 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:46:15.364759 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:15.365906 (+ 1147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:15.367387 (+ 1481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:15.367827 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:15.367930 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:15.371570 (+ 3640us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:46:15.371581 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:46:15.371591 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:46:15.371622 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:46:15.375248 (+ 3626us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:15.375954 (+ 706us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:15.375962 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:15.375970 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:15.376183 (+ 213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:15.376629 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:15.376633 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:15.376635 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:15.377258 (+ 623us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:46:15.377446 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0619 16:46:15.378861 (+ 1415us) server_negotiation.cc:299] Negotiation successful
0619 16:46:15.379065 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":648,"thread_start_us":483,"threads_started":1}
I20260619 16:46:15.380009 14029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:15.357385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40434 (local address 127.0.97.60:33823)
0619 16:46:15.358171 (+ 786us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:15.358186 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:15.358234 (+ 48us) server_negotiation.cc:407] Connection header received
0619 16:46:15.359981 (+ 1747us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:15.360006 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:15.360505 (+ 499us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:15.360914 (+ 409us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:46:15.362130 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:15.364178 (+ 2048us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:15.365597 (+ 1419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:15.366497 (+ 900us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:15.366669 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:15.372427 (+ 5758us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:46:15.372434 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:46:15.372438 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:46:15.372487 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:46:15.375718 (+ 3231us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:15.376443 (+ 725us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:15.376447 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:15.376451 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:15.376566 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:15.377013 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:15.377017 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:15.377024 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:15.377338 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:46:15.378286 (+ 948us) server_negotiation.cc:1035] Waiting for connection context
0619 16:46:15.378513 (+ 227us) server_negotiation.cc:299] Negotiation successful
0619 16:46:15.378909 (+ 396us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":556,"thread_start_us":333,"threads_started":1}
I20260619 16:46:15.385531 14031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:15.369119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41079 (local address 127.0.0.1:56902)
0619 16:46:15.369816 (+ 697us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:15.369865 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0619 16:46:15.370067 (+ 202us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:46:15.370426 (+ 359us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:46:15.370434 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:46:15.371203 (+ 769us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:46:15.371608 (+ 405us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:15.371622 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:15.373033 (+ 1411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:15.373039 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:15.374641 (+ 1602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:15.374656 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:15.375668 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:15.375675 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:15.375846 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:15.376385 (+ 539us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:46:15.376404 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:46:15.379111 (+ 2707us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:46:15.382577 (+ 3466us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:46:15.382587 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:46:15.382594 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:46:15.383084 (+ 490us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:46:15.383529 (+ 445us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:46:15.383534 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:46:15.383537 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:46:15.383676 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:46:15.384465 (+ 789us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:46:15.384472 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:46:15.384652 (+ 180us) client_negotiation.cc:769] Sending connection context
0619 16:46:15.384833 (+ 181us) client_negotiation.cc:240] Negotiation successful
0619 16:46:15.385138 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":464,"thread_start_us":339,"threads_started":1}
I20260619 16:46:15.385598 13880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:15.369088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56902 (local address 127.0.97.62:41079)
0619 16:46:15.369268 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:15.369273 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:15.369951 (+ 678us) server_negotiation.cc:407] Connection header received
0619 16:46:15.370192 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:15.370198 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:15.370262 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:15.370335 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:46:15.371753 (+ 1418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:15.372834 (+ 1081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:15.374814 (+ 1980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:15.375461 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:15.375583 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:15.379298 (+ 3715us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:46:15.379305 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:46:15.379310 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:46:15.379345 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:46:15.382432 (+ 3087us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:15.383217 (+ 785us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:15.383222 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:15.383226 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:15.383391 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:15.383803 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:15.383809 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:15.383813 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:15.384333 (+ 520us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:46:15.384510 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0619 16:46:15.384965 (+ 455us) server_negotiation.cc:299] Negotiation successful
0619 16:46:15.385189 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260619 16:46:15.386891 13878 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6dd438a401754d518fa33f2f1b349330: Bootstrap starting.
I20260619 16:46:15.389810 14026 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38025 } has no permanent_uuid. Determining permanent_uuid...
I20260619 16:46:15.390336 13878 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6dd438a401754d518fa33f2f1b349330: Neither blocks nor log segments found. Creating new log.
I20260619 16:46:15.391608 13878 log.cc:824] T 00000000000000000000000000000000 P 6dd438a401754d518fa33f2f1b349330: Log is configured to *not* fsync() on all Append() calls
I20260619 16:46:15.394712 13878 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6dd438a401754d518fa33f2f1b349330: No bootstrap required, opened a new log
Jun 19 16:46:15 dist-test-slave-wvz7 krb5kdc[13798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887573, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
Jun 19 16:46:15 dist-test-slave-wvz7 krb5kdc[13798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887575, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260619 16:46:15.401949 14031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:15.390392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:38025 (local address 127.0.0.1:41406)
0619 16:46:15.390659 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:15.390674 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 16:46:15.390774 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:46:15.391071 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:46:15.391075 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:46:15.391408 (+ 333us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:46:15.391542 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:15.391550 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:15.392757 (+ 1207us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:15.392762 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:15.393626 (+ 864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:15.393638 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:15.394323 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:15.394329 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:15.394496 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:15.394941 (+ 445us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:46:15.394956 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:46:15.396845 (+ 1889us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:46:15.399612 (+ 2767us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:46:15.399618 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:46:15.399622 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:46:15.399976 (+ 354us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:46:15.400312 (+ 336us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:46:15.400317 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:46:15.400321 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:46:15.400449 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:46:15.401080 (+ 631us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:46:15.401085 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:46:15.401226 (+ 141us) client_negotiation.cc:769] Sending connection context
0619 16:46:15.401360 (+ 134us) client_negotiation.cc:240] Negotiation successful
0619 16:46:15.401614 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157}
I20260619 16:46:15.402362 13957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:15.390492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41406 (local address 127.0.97.61:38025)
0619 16:46:15.390651 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:15.390657 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:15.390733 (+ 76us) server_negotiation.cc:407] Connection header received
0619 16:46:15.390886 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:15.390891 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:15.390962 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:15.391045 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:46:15.391671 (+ 626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:15.392615 (+ 944us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:15.393786 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:15.394189 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:15.394292 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:15.397030 (+ 2738us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:46:15.397034 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:46:15.397037 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:46:15.397070 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:46:15.399496 (+ 2426us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:15.400080 (+ 584us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:15.400084 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:15.400086 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:15.400213 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:15.400542 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:15.400547 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:15.400550 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:15.400880 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:46:15.401017 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0619 16:46:15.401464 (+ 447us) server_negotiation.cc:299] Negotiation successful
0619 16:46:15.401659 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260619 16:46:15.403053 13880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:15.383084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56914 (local address 127.0.97.62:41079)
0619 16:46:15.385823 (+ 2739us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:15.385828 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:15.385846 (+ 18us) server_negotiation.cc:407] Connection header received
0619 16:46:15.385892 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:15.385897 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:15.385967 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:15.387662 (+ 1695us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:46:15.387792 (+ 130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:15.389030 (+ 1238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:15.390935 (+ 1905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:15.391582 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:15.391805 (+ 223us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:15.396476 (+ 4671us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:46:15.396483 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:46:15.396487 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:46:15.396534 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:46:15.399921 (+ 3387us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:15.400670 (+ 749us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:15.400675 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:15.400678 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:15.400823 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:15.401287 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:15.401292 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:15.401295 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:15.401657 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:46:15.401962 (+ 305us) server_negotiation.cc:1035] Waiting for connection context
0619 16:46:15.402449 (+ 487us) server_negotiation.cc:299] Negotiation successful
0619 16:46:15.402724 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2619,"thread_start_us":5695,"threads_started":1}
I20260619 16:46:15.403057 14026 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33823 } has no permanent_uuid. Determining permanent_uuid...
I20260619 16:46:15.405704 13878 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6dd438a401754d518fa33f2f1b349330 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dd438a401754d518fa33f2f1b349330" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41079 } } peers { permanent_uuid: "f4593c435c314505ac60b4a5aa3eebe3" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38025 } } peers { permanent_uuid: "34bdd0b575de4d37b6d6c5760c7bc982" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33823 } }
I20260619 16:46:15.406071 13878 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6dd438a401754d518fa33f2f1b349330 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:46:15.406191 13878 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6dd438a401754d518fa33f2f1b349330 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6dd438a401754d518fa33f2f1b349330, State: Initialized, Role: FOLLOWER
I20260619 16:46:15.406770 13878 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6dd438a401754d518fa33f2f1b349330 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dd438a401754d518fa33f2f1b349330" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41079 } } peers { permanent_uuid: "f4593c435c314505ac60b4a5aa3eebe3" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38025 } } peers { permanent_uuid: "34bdd0b575de4d37b6d6c5760c7bc982" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33823 } }
I20260619 16:46:15.407634 14034 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6dd438a401754d518fa33f2f1b349330 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dd438a401754d518fa33f2f1b349330" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41079 } } peers { permanent_uuid: "f4593c435c314505ac60b4a5aa3eebe3" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38025 } } peers { permanent_uuid: "34bdd0b575de4d37b6d6c5760c7bc982" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33823 } } }
I20260619 16:46:15.408083 14034 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6dd438a401754d518fa33f2f1b349330 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 16:46:15.408841 13878 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6dd438a401754d518fa33f2f1b349330 [sys.catalog]: configured and running, proceeding with master startup.
Jun 19 16:46:15 dist-test-slave-wvz7 krb5kdc[13798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887575, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
Jun 19 16:46:15 dist-test-slave-wvz7 krb5kdc[13798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887573, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260619 16:46:15.417315 14031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:15.403631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:33823 (local address 127.0.0.1:40450)
0619 16:46:15.403758 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:15.403775 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 16:46:15.403859 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:46:15.404305 (+ 446us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:46:15.404309 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:46:15.404716 (+ 407us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:46:15.404886 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:15.404897 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:15.406148 (+ 1251us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:15.406153 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:15.407382 (+ 1229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:15.407396 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:15.408356 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:15.408361 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:15.408511 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:15.409028 (+ 517us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:46:15.409047 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:46:15.411465 (+ 2418us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:46:15.414820 (+ 3355us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:46:15.414824 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:46:15.414827 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:46:15.415269 (+ 442us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:46:15.415665 (+ 396us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:46:15.415668 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:46:15.415671 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:46:15.415805 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:46:15.416435 (+ 630us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:46:15.416439 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:46:15.416579 (+ 140us) client_negotiation.cc:769] Sending connection context
0619 16:46:15.416728 (+ 149us) client_negotiation.cc:240] Negotiation successful
0619 16:46:15.416997 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38}
I20260619 16:46:15.418200 14029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:15.403721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40450 (local address 127.0.97.60:33823)
0619 16:46:15.403842 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:15.403846 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:15.403861 (+ 15us) server_negotiation.cc:407] Connection header received
0619 16:46:15.403971 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:15.403976 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:15.404077 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:15.404158 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:46:15.405024 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:15.406012 (+ 988us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:15.407523 (+ 1511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:15.408099 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:15.408206 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:15.411657 (+ 3451us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:46:15.411663 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:46:15.411667 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:46:15.411701 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:46:15.414709 (+ 3008us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:15.415393 (+ 684us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:15.415398 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:15.415401 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:15.415541 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:15.415916 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:15.415933 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:15.415937 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:15.416270 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:46:15.416392 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0619 16:46:15.417662 (+ 1270us) server_negotiation.cc:299] Negotiation successful
0619 16:46:15.417893 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260619 16:46:15.420657 13957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:15.405399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41416 (local address 127.0.97.61:38025)
0619 16:46:15.405544 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:15.405549 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:15.405563 (+ 14us) server_negotiation.cc:407] Connection header received
0619 16:46:15.405602 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:15.405605 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:15.405650 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:15.405722 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:46:15.406994 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:15.407945 (+ 951us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:15.409370 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:15.409800 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:15.409904 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:15.414806 (+ 4902us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:46:15.414812 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:46:15.414816 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:46:15.414850 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:46:15.417997 (+ 3147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:15.418948 (+ 951us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:15.418954 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:15.418958 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:15.419076 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:15.419421 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:15.419426 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:15.419430 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:15.419665 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:46:15.419905 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0619 16:46:15.420187 (+ 282us) server_negotiation.cc:299] Negotiation successful
0619 16:46:15.420371 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
W20260619 16:46:15.424415 14045 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 6dd438a401754d518fa33f2f1b349330: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 16:46:15.424549 14045 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 16:46:15.426333 14026 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 34bdd0b575de4d37b6d6c5760c7bc982: Bootstrap starting.
I20260619 16:46:15.430182 14026 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 34bdd0b575de4d37b6d6c5760c7bc982: Neither blocks nor log segments found. Creating new log.
I20260619 16:46:15.430456 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 16:46:15.431341 14026 log.cc:824] T 00000000000000000000000000000000 P 34bdd0b575de4d37b6d6c5760c7bc982: Log is configured to *not* fsync() on all Append() calls
I20260619 16:46:15.427825 13811 master_runner.cc:186] Error getting master registration for 127.0.97.61:38025: OK, instance_id { permanent_uuid: "f4593c435c314505ac60b4a5aa3eebe3" instance_seqno: 1781887574880232 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260619 16:46:15.438835 13811 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260619 16:46:15.440549 14026 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 34bdd0b575de4d37b6d6c5760c7bc982: No bootstrap required, opened a new log
I20260619 16:46:15.450336 14026 raft_consensus.cc:348] T 00000000000000000000000000000000 P 34bdd0b575de4d37b6d6c5760c7bc982 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dd438a401754d518fa33f2f1b349330" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41079 } } peers { permanent_uuid: "f4593c435c314505ac60b4a5aa3eebe3" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38025 } } peers { permanent_uuid: "34bdd0b575de4d37b6d6c5760c7bc982" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33823 } }
I20260619 16:46:15.450770 14026 raft_consensus.cc:374] T 00000000000000000000000000000000 P 34bdd0b575de4d37b6d6c5760c7bc982 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:46:15.450878 14026 raft_consensus.cc:729] T 00000000000000000000000000000000 P 34bdd0b575de4d37b6d6c5760c7bc982 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 34bdd0b575de4d37b6d6c5760c7bc982, State: Initialized, Role: FOLLOWER
I20260619 16:46:15.451629 14026 consensus_queue.cc:260] T 00000000000000000000000000000000 P 34bdd0b575de4d37b6d6c5760c7bc982 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dd438a401754d518fa33f2f1b349330" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41079 } } peers { permanent_uuid: "f4593c435c314505ac60b4a5aa3eebe3" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38025 } } peers { permanent_uuid: "34bdd0b575de4d37b6d6c5760c7bc982" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33823 } }
I20260619 16:46:15.456151 14054 sys_catalog.cc:455] T 00000000000000000000000000000000 P 34bdd0b575de4d37b6d6c5760c7bc982 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dd438a401754d518fa33f2f1b349330" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41079 } } peers { permanent_uuid: "f4593c435c314505ac60b4a5aa3eebe3" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38025 } } peers { permanent_uuid: "34bdd0b575de4d37b6d6c5760c7bc982" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33823 } } }
I20260619 16:46:15.456558 14054 sys_catalog.cc:458] T 00000000000000000000000000000000 P 34bdd0b575de4d37b6d6c5760c7bc982 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 16:46:15.458364 14026 sys_catalog.cc:564] T 00000000000000000000000000000000 P 34bdd0b575de4d37b6d6c5760c7bc982 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:46:15.480909 13880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:15.448183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56920 (local address 127.0.97.62:41079)
0619 16:46:15.448348 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:15.448355 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:15.461121 (+ 12766us) server_negotiation.cc:407] Connection header received
0619 16:46:15.461867 (+ 746us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:15.461873 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:15.461957 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:15.462026 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:46:15.464233 (+ 2207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:15.465318 (+ 1085us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:15.467471 (+ 2153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:15.468156 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:15.468254 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:15.471962 (+ 3708us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:46:15.471969 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:46:15.471975 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:46:15.472015 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:46:15.475900 (+ 3885us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:15.476952 (+ 1052us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:15.476958 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:15.476962 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:15.477143 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:15.477582 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:15.477588 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:15.477591 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:15.477956 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:46:15.478216 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0619 16:46:15.479197 (+ 981us) server_negotiation.cc:299] Negotiation successful
0619 16:46:15.479437 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260619 16:46:15.481406 13957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:15.448234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41428 (local address 127.0.97.61:38025)
0619 16:46:15.448429 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:15.448435 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:15.455596 (+ 7161us) server_negotiation.cc:407] Connection header received
0619 16:46:15.459849 (+ 4253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:15.459857 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:15.459949 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:15.460023 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:46:15.462878 (+ 2855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:15.464065 (+ 1187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:15.468649 (+ 4584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:15.469361 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:15.469456 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:15.474185 (+ 4729us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:46:15.474193 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:46:15.474199 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:46:15.474246 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:46:15.477647 (+ 3401us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:15.478626 (+ 979us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:15.478633 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:15.478637 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:15.478782 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:15.479220 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:15.479226 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:15.479229 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:15.479599 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:46:15.479872 (+ 273us) server_negotiation.cc:1035] Waiting for connection context
0619 16:46:15.480736 (+ 864us) server_negotiation.cc:299] Negotiation successful
0619 16:46:15.481045 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260619 16:46:15.498718 14029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:15.455761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40460 (local address 127.0.97.60:33823)
0619 16:46:15.455943 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:15.455949 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:15.482269 (+ 26320us) server_negotiation.cc:407] Connection header received
0619 16:46:15.482629 (+ 360us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:15.482635 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:15.482715 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:15.482784 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:46:15.484653 (+ 1869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:15.485711 (+ 1058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:15.487584 (+ 1873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:15.488362 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:15.488492 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:15.491560 (+ 3068us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:46:15.491570 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:46:15.491576 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:46:15.491617 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:46:15.495085 (+ 3468us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:15.495870 (+ 785us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:15.495875 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:15.495878 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:15.496021 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:15.496435 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:15.496440 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:15.496443 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:15.496806 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:46:15.498040 (+ 1234us) server_negotiation.cc:1035] Waiting for connection context
0619 16:46:15.498159 (+ 119us) server_negotiation.cc:299] Negotiation successful
0619 16:46:15.498362 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
W20260619 16:46:15.506124 14066 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 34bdd0b575de4d37b6d6c5760c7bc982: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 16:46:15.506248 14066 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 16:46:15.498234 13959 master_runner.cc:186] Error getting master registration for 127.0.97.61:38025: OK, instance_id { permanent_uuid: "f4593c435c314505ac60b4a5aa3eebe3" instance_seqno: 1781887574880232 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260619 16:46:15.508473 13959 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260619 16:46:15.587957 13980 tablet.cc:2369] T 00000000000000000000000000000000 P 34bdd0b575de4d37b6d6c5760c7bc982: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 16:46:15.599476 13952 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33823 } attempt: 5
Jun 19 16:46:15 dist-test-slave-wvz7 krb5kdc[13798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887574, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
I20260619 16:46:15.616297 13955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:15.600166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:33823 (local address 127.0.0.1:40476)
0619 16:46:15.600353 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:15.600372 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 16:46:15.600486 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:46:15.600852 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:46:15.600857 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:46:15.601382 (+ 525us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:46:15.601605 (+ 223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:15.601618 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:15.603210 (+ 1592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:15.603216 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:15.604697 (+ 1481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:15.604712 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:15.605671 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:15.605678 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:15.605920 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:15.606560 (+ 640us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:46:15.606584 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:46:15.609431 (+ 2847us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:46:15.613530 (+ 4099us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:46:15.613536 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:46:15.613541 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:46:15.614109 (+ 568us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:46:15.614531 (+ 422us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:46:15.614538 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:46:15.614541 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:46:15.614703 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:46:15.615309 (+ 606us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:46:15.615314 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:46:15.615483 (+ 169us) client_negotiation.cc:769] Sending connection context
0619 16:46:15.615639 (+ 156us) client_negotiation.cc:240] Negotiation successful
0619 16:46:15.615950 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":62}
I20260619 16:46:15.616345 14029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:15.600247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40476 (local address 127.0.97.60:33823)
0619 16:46:15.600424 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:15.600430 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:15.600450 (+ 20us) server_negotiation.cc:407] Connection header received
0619 16:46:15.600623 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:15.600628 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:15.600705 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:15.600811 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:46:15.601778 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:15.603062 (+ 1284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:15.604831 (+ 1769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:15.605510 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:15.605638 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:15.609624 (+ 3986us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:46:15.609631 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:46:15.609637 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:46:15.609676 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:46:15.613366 (+ 3690us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:15.614225 (+ 859us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:15.614231 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:15.614235 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:15.614393 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:15.614810 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:15.614815 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:15.614819 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:15.615198 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:46:15.615343 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0619 16:46:15.615750 (+ 407us) server_negotiation.cc:299] Negotiation successful
0619 16:46:15.615973 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260619 16:46:15.622627 13952 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f4593c435c314505ac60b4a5aa3eebe3: Bootstrap starting.
I20260619 16:46:15.626672 13952 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f4593c435c314505ac60b4a5aa3eebe3: Neither blocks nor log segments found. Creating new log.
I20260619 16:46:15.627948 13952 log.cc:824] T 00000000000000000000000000000000 P f4593c435c314505ac60b4a5aa3eebe3: Log is configured to *not* fsync() on all Append() calls
I20260619 16:46:15.631668 13952 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f4593c435c314505ac60b4a5aa3eebe3: No bootstrap required, opened a new log
I20260619 16:46:15.642313 13952 raft_consensus.cc:348] T 00000000000000000000000000000000 P f4593c435c314505ac60b4a5aa3eebe3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dd438a401754d518fa33f2f1b349330" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41079 } } peers { permanent_uuid: "f4593c435c314505ac60b4a5aa3eebe3" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38025 } } peers { permanent_uuid: "34bdd0b575de4d37b6d6c5760c7bc982" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33823 } }
I20260619 16:46:15.642729 13952 raft_consensus.cc:374] T 00000000000000000000000000000000 P f4593c435c314505ac60b4a5aa3eebe3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:46:15.642844 13952 raft_consensus.cc:729] T 00000000000000000000000000000000 P f4593c435c314505ac60b4a5aa3eebe3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4593c435c314505ac60b4a5aa3eebe3, State: Initialized, Role: FOLLOWER
I20260619 16:46:15.643522 13952 consensus_queue.cc:260] T 00000000000000000000000000000000 P f4593c435c314505ac60b4a5aa3eebe3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dd438a401754d518fa33f2f1b349330" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41079 } } peers { permanent_uuid: "f4593c435c314505ac60b4a5aa3eebe3" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38025 } } peers { permanent_uuid: "34bdd0b575de4d37b6d6c5760c7bc982" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33823 } }
I20260619 16:46:15.644578 14069 sys_catalog.cc:455] T 00000000000000000000000000000000 P f4593c435c314505ac60b4a5aa3eebe3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dd438a401754d518fa33f2f1b349330" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41079 } } peers { permanent_uuid: "f4593c435c314505ac60b4a5aa3eebe3" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38025 } } peers { permanent_uuid: "34bdd0b575de4d37b6d6c5760c7bc982" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33823 } } }
I20260619 16:46:15.645030 14069 sys_catalog.cc:458] T 00000000000000000000000000000000 P f4593c435c314505ac60b4a5aa3eebe3 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 16:46:15.645911 13952 sys_catalog.cc:564] T 00000000000000000000000000000000 P f4593c435c314505ac60b4a5aa3eebe3 [sys.catalog]: configured and running, proceeding with master startup.
W20260619 16:46:15.653164 13832 tablet.cc:2369] T 00000000000000000000000000000000 P 6dd438a401754d518fa33f2f1b349330: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 16:46:15.671808 13885 master_runner.cc:263] No leader master found from master f4593c435c314505ac60b4a5aa3eebe3
I20260619 16:46:15.671977 13885 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260619 16:46:15.674682 14080 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f4593c435c314505ac60b4a5aa3eebe3: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 16:46:15.674817 14080 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 16:46:15.777302 14054 raft_consensus.cc:482] T 00000000000000000000000000000000 P 34bdd0b575de4d37b6d6c5760c7bc982 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 16:46:15.777669 14054 raft_consensus.cc:504] T 00000000000000000000000000000000 P 34bdd0b575de4d37b6d6c5760c7bc982 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dd438a401754d518fa33f2f1b349330" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41079 } } peers { permanent_uuid: "f4593c435c314505ac60b4a5aa3eebe3" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38025 } } peers { permanent_uuid: "34bdd0b575de4d37b6d6c5760c7bc982" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33823 } }
I20260619 16:46:15.779424 14054 leader_election.cc:288] T 00000000000000000000000000000000 P 34bdd0b575de4d37b6d6c5760c7bc982 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6dd438a401754d518fa33f2f1b349330 (127.0.97.62:41079), f4593c435c314505ac60b4a5aa3eebe3 (127.0.97.61:38025)
I20260619 16:46:15.780465 13853 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "34bdd0b575de4d37b6d6c5760c7bc982" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6dd438a401754d518fa33f2f1b349330" is_pre_election: true
I20260619 16:46:15.781023 13853 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 6dd438a401754d518fa33f2f1b349330 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 34bdd0b575de4d37b6d6c5760c7bc982 in term 0.
I20260619 16:46:15.781663 13978 leader_election.cc:302] T 00000000000000000000000000000000 P 34bdd0b575de4d37b6d6c5760c7bc982 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 34bdd0b575de4d37b6d6c5760c7bc982, 6dd438a401754d518fa33f2f1b349330; no voters:
I20260619 16:46:15.782851 13927 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "34bdd0b575de4d37b6d6c5760c7bc982" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4593c435c314505ac60b4a5aa3eebe3" is_pre_election: true
I20260619 16:46:15.783447 13927 raft_consensus.cc:2454] T 00000000000000000000000000000000 P f4593c435c314505ac60b4a5aa3eebe3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 34bdd0b575de4d37b6d6c5760c7bc982 in term 0.
I20260619 16:46:15.784297 14054 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 34bdd0b575de4d37b6d6c5760c7bc982 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 16:46:15.784472 14054 raft_consensus.cc:482] T 00000000000000000000000000000000 P 34bdd0b575de4d37b6d6c5760c7bc982 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 16:46:15.784588 14054 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 34bdd0b575de4d37b6d6c5760c7bc982 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:46:15.787935 14054 raft_consensus.cc:504] T 00000000000000000000000000000000 P 34bdd0b575de4d37b6d6c5760c7bc982 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dd438a401754d518fa33f2f1b349330" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41079 } } peers { permanent_uuid: "f4593c435c314505ac60b4a5aa3eebe3" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38025 } } peers { permanent_uuid: "34bdd0b575de4d37b6d6c5760c7bc982" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33823 } }
I20260619 16:46:15.789047 13853 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "34bdd0b575de4d37b6d6c5760c7bc982" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6dd438a401754d518fa33f2f1b349330"
I20260619 16:46:15.789299 13853 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6dd438a401754d518fa33f2f1b349330 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:46:15.797720 13853 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 6dd438a401754d518fa33f2f1b349330 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 34bdd0b575de4d37b6d6c5760c7bc982 in term 1.
I20260619 16:46:15.798197 13978 leader_election.cc:302] T 00000000000000000000000000000000 P 34bdd0b575de4d37b6d6c5760c7bc982 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 34bdd0b575de4d37b6d6c5760c7bc982, 6dd438a401754d518fa33f2f1b349330; no voters:
I20260619 16:46:15.802989 13927 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "34bdd0b575de4d37b6d6c5760c7bc982" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4593c435c314505ac60b4a5aa3eebe3"
I20260619 16:46:15.803241 13927 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f4593c435c314505ac60b4a5aa3eebe3 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:46:15.806670 13927 raft_consensus.cc:2454] T 00000000000000000000000000000000 P f4593c435c314505ac60b4a5aa3eebe3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 34bdd0b575de4d37b6d6c5760c7bc982 in term 1.
I20260619 16:46:15.807331 14054 leader_election.cc:288] T 00000000000000000000000000000000 P 34bdd0b575de4d37b6d6c5760c7bc982 [CANDIDATE]: Term 1 election: Requested vote from peers 6dd438a401754d518fa33f2f1b349330 (127.0.97.62:41079), f4593c435c314505ac60b4a5aa3eebe3 (127.0.97.61:38025)
I20260619 16:46:15.807557 14054 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 34bdd0b575de4d37b6d6c5760c7bc982 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:46:15.807698 14054 raft_consensus.cc:686] T 00000000000000000000000000000000 P 34bdd0b575de4d37b6d6c5760c7bc982 [term 1 LEADER]: Becoming Leader. State: Replica: 34bdd0b575de4d37b6d6c5760c7bc982, State: Running, Role: LEADER
I20260619 16:46:15.808214 14054 consensus_queue.cc:237] T 00000000000000000000000000000000 P 34bdd0b575de4d37b6d6c5760c7bc982 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dd438a401754d518fa33f2f1b349330" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41079 } } peers { permanent_uuid: "f4593c435c314505ac60b4a5aa3eebe3" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38025 } } peers { permanent_uuid: "34bdd0b575de4d37b6d6c5760c7bc982" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33823 } }
I20260619 16:46:15.818197 14054 sys_catalog.cc:455] T 00000000000000000000000000000000 P 34bdd0b575de4d37b6d6c5760c7bc982 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 34bdd0b575de4d37b6d6c5760c7bc982. Latest consensus state: current_term: 1 leader_uuid: "34bdd0b575de4d37b6d6c5760c7bc982" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dd438a401754d518fa33f2f1b349330" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41079 } } peers { permanent_uuid: "f4593c435c314505ac60b4a5aa3eebe3" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38025 } } peers { permanent_uuid: "34bdd0b575de4d37b6d6c5760c7bc982" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33823 } } }
I20260619 16:46:15.818523 14054 sys_catalog.cc:458] T 00000000000000000000000000000000 P 34bdd0b575de4d37b6d6c5760c7bc982 [sys.catalog]: This master's current role is: LEADER
I20260619 16:46:15.820132 14084 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:46:15.827462 14084 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:46:15.842720 13927 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f4593c435c314505ac60b4a5aa3eebe3 [term 1 FOLLOWER]: Refusing update from remote peer 34bdd0b575de4d37b6d6c5760c7bc982: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:46:15.843797 14054 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 34bdd0b575de4d37b6d6c5760c7bc982 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f4593c435c314505ac60b4a5aa3eebe3" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:46:15.854344 13853 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 6dd438a401754d518fa33f2f1b349330 [term 1 FOLLOWER]: Refusing update from remote peer 34bdd0b575de4d37b6d6c5760c7bc982: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:46:15.855383 14054 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 34bdd0b575de4d37b6d6c5760c7bc982 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6dd438a401754d518fa33f2f1b349330" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41079 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260619 16:46:15.871220 13906 tablet.cc:2369] T 00000000000000000000000000000000 P f4593c435c314505ac60b4a5aa3eebe3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 16:46:15.912876 14034 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6dd438a401754d518fa33f2f1b349330 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 34bdd0b575de4d37b6d6c5760c7bc982. Latest consensus state: current_term: 1 leader_uuid: "34bdd0b575de4d37b6d6c5760c7bc982" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dd438a401754d518fa33f2f1b349330" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41079 } } peers { permanent_uuid: "f4593c435c314505ac60b4a5aa3eebe3" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38025 } } peers { permanent_uuid: "34bdd0b575de4d37b6d6c5760c7bc982" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33823 } } }
I20260619 16:46:15.913216 14034 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6dd438a401754d518fa33f2f1b349330 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 16:46:15.915974 14085 mvcc.cc:205] Tried to move back new op lower bound from 7298611510641602560 to 7298611510524870656. Current Snapshot: MvccSnapshot[applied={T|T < 7298611510641602560}]
I20260619 16:46:15.920197 14087 mvcc.cc:205] Tried to move back new op lower bound from 7298611510641602560 to 7298611510524870656. Current Snapshot: MvccSnapshot[applied={T|T < 7298611510641602560}]
I20260619 16:46:15.922262 14034 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6dd438a401754d518fa33f2f1b349330 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "34bdd0b575de4d37b6d6c5760c7bc982" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dd438a401754d518fa33f2f1b349330" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41079 } } peers { permanent_uuid: "f4593c435c314505ac60b4a5aa3eebe3" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38025 } } peers { permanent_uuid: "34bdd0b575de4d37b6d6c5760c7bc982" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33823 } } }
I20260619 16:46:15.922506 14034 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6dd438a401754d518fa33f2f1b349330 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 16:46:15.926522 14069 sys_catalog.cc:455] T 00000000000000000000000000000000 P f4593c435c314505ac60b4a5aa3eebe3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 34bdd0b575de4d37b6d6c5760c7bc982. Latest consensus state: current_term: 1 leader_uuid: "34bdd0b575de4d37b6d6c5760c7bc982" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dd438a401754d518fa33f2f1b349330" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41079 } } peers { permanent_uuid: "f4593c435c314505ac60b4a5aa3eebe3" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38025 } } peers { permanent_uuid: "34bdd0b575de4d37b6d6c5760c7bc982" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33823 } } }
I20260619 16:46:15.926831 14069 sys_catalog.cc:458] T 00000000000000000000000000000000 P f4593c435c314505ac60b4a5aa3eebe3 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 16:46:15.934087 14082 sys_catalog.cc:455] T 00000000000000000000000000000000 P 34bdd0b575de4d37b6d6c5760c7bc982 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "34bdd0b575de4d37b6d6c5760c7bc982" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dd438a401754d518fa33f2f1b349330" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41079 } } peers { permanent_uuid: "f4593c435c314505ac60b4a5aa3eebe3" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38025 } } peers { permanent_uuid: "34bdd0b575de4d37b6d6c5760c7bc982" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33823 } } }
I20260619 16:46:15.934422 14082 sys_catalog.cc:458] T 00000000000000000000000000000000 P 34bdd0b575de4d37b6d6c5760c7bc982 [sys.catalog]: This master's current role is: LEADER
I20260619 16:46:15.943138 14084 catalog_manager.cc:1347] Generated new cluster ID: 92de685544d049e298dbc22d13e3cf0c
I20260619 16:46:15.943243 14084 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:46:15.951337 14082 sys_catalog.cc:455] T 00000000000000000000000000000000 P 34bdd0b575de4d37b6d6c5760c7bc982 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "34bdd0b575de4d37b6d6c5760c7bc982" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dd438a401754d518fa33f2f1b349330" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41079 } } peers { permanent_uuid: "f4593c435c314505ac60b4a5aa3eebe3" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38025 } } peers { permanent_uuid: "34bdd0b575de4d37b6d6c5760c7bc982" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33823 } } }
I20260619 16:46:15.951608 14082 sys_catalog.cc:458] T 00000000000000000000000000000000 P 34bdd0b575de4d37b6d6c5760c7bc982 [sys.catalog]: This master's current role is: LEADER
I20260619 16:46:15.955080 14086 mvcc.cc:205] Tried to move back new op lower bound from 7298611510641602560 to 7298611510524870656. Current Snapshot: MvccSnapshot[applied={T|T < 7298611510641602560}]
I20260619 16:46:15.961982 14069 sys_catalog.cc:455] T 00000000000000000000000000000000 P f4593c435c314505ac60b4a5aa3eebe3 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "34bdd0b575de4d37b6d6c5760c7bc982" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dd438a401754d518fa33f2f1b349330" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41079 } } peers { permanent_uuid: "f4593c435c314505ac60b4a5aa3eebe3" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38025 } } peers { permanent_uuid: "34bdd0b575de4d37b6d6c5760c7bc982" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33823 } } }
I20260619 16:46:15.962230 14069 sys_catalog.cc:458] T 00000000000000000000000000000000 P f4593c435c314505ac60b4a5aa3eebe3 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 16:46:15.977581 14084 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 16:46:15.979916 14084 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:46:15.993091 14084 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 34bdd0b575de4d37b6d6c5760c7bc982: Generated new TSK 0
I20260619 16:46:15.994196 14084 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 16:46:16.426903 14045 catalog_manager.cc:1259] Loaded cluster ID: 92de685544d049e298dbc22d13e3cf0c
I20260619 16:46:16.427019 14045 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 6dd438a401754d518fa33f2f1b349330: loading cluster ID for follower catalog manager: success
I20260619 16:46:16.430476 14045 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 6dd438a401754d518fa33f2f1b349330: acquiring CA information for follower catalog manager: success
I20260619 16:46:16.432579 14045 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 6dd438a401754d518fa33f2f1b349330: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 16:46:16.676652 14080 catalog_manager.cc:1259] Loaded cluster ID: 92de685544d049e298dbc22d13e3cf0c
I20260619 16:46:16.676764 14080 catalog_manager.cc:1552] T 00000000000000000000000000000000 P f4593c435c314505ac60b4a5aa3eebe3: loading cluster ID for follower catalog manager: success
I20260619 16:46:16.679164 14080 catalog_manager.cc:1574] T 00000000000000000000000000000000 P f4593c435c314505ac60b4a5aa3eebe3: acquiring CA information for follower catalog manager: success
I20260619 16:46:16.680859 14080 catalog_manager.cc:1602] T 00000000000000000000000000000000 P f4593c435c314505ac60b4a5aa3eebe3: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 16:46:17.089169 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu pid 13811
I20260619 16:46:17.090341 13873 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:46:17.199110 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu with pid 13811
I20260619 16:46:17.217061 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu pid 13885
I20260619 16:46:17.218073 13947 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:46:17.321921 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu with pid 13885
I20260619 16:46:17.337577 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu pid 13959
I20260619 16:46:17.338626 14021 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:46:17.444717 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu with pid 13959
2026-06-19T16:46:17Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (4632 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781887385903558-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 16:46:17 dist-test-slave-wvz7 krb5kdc[14102](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 16:46:17 dist-test-slave-wvz7 krb5kdc[14102](info): set up 2 sockets
Jun 19 16:46:17 dist-test-slave-wvz7 krb5kdc[14102](info): commencing operation
krb5kdc: starting...
W20260619 16:46:18.453878 388 mini_kdc.cc:121] Time spent starting KDC: real 0.973s user 0.000s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781887385903558-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781887385903558-388-0/krb5kdc/test-user.keytab.
Jun 19 16:46:18 dist-test-slave-wvz7 krb5kdc[14102](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887578, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T16:46:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T16:46:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781887385903558-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781887385903558-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260619 16:46:18.651412 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu
/tmp/dist-test-taskEs2EH8/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781887385903558-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781887385903558-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781887385903558-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781887385903558-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:38527
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781887385903558-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36667
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:38527
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781887385903558-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781887385903558-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781887385903558-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:46:18.955832 14115 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:46:18.956115 14115 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:46:18.956210 14115 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:46:18.963332 14115 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:46:18.963439 14115 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:46:18.963483 14115 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:46:18.963541 14115 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:46:18.963582 14115 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:46:18.973268 14115 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36667
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781887385903558-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781887385903558-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:38527
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781887385903558-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:38527
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781887385903558-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781887385903558-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:46:18.974531 14115 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:46:18.975719 14115 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:46:18.986899 14120 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:46:18.986871 14121 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:46:18.986899 14123 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:46:18.987459 14115 server_base.cc:1034] running on GCE node
I20260619 16:46:18.988327 14115 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:46:18.989748 14115 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:46:18.990965 14115 hybrid_clock.cc:648] HybridClock initialized: now 1781887578990915 us; error 42 us; skew 500 ppm
I20260619 16:46:18.991299 14115 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:46:18 dist-test-slave-wvz7 krb5kdc[14102](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887578, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:46:18.994294 14115 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260619 16:46:18.995656 14115 webserver.cc:460] Webserver started at http://127.0.97.62:38677/ using document root <none> and password file <none>
I20260619 16:46:18.996325 14115 fs_manager.cc:359] Metadata directory not provided
I20260619 16:46:18.996435 14115 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:46:18.996718 14115 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:46:18.999574 14115 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781887385903558-388-0/minicluster-data/master-0/data/instance:
uuid: "895a80903ac44b9ca0f30c9f12905bc5"
format_stamp: "Formatted at 2026-06-19 16:46:18 on dist-test-slave-wvz7"
server_key: "0b87e94c2612f0f062136acaa9e1ed4c"
server_key_iv: "337812299c40116c297b317fcf4fa0ad"
server_key_version: "encryptionkey@0"
I20260619 16:46:19.000308 14115 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781887385903558-388-0/minicluster-data/master-0/wal/instance:
uuid: "895a80903ac44b9ca0f30c9f12905bc5"
format_stamp: "Formatted at 2026-06-19 16:46:18 on dist-test-slave-wvz7"
server_key: "0b87e94c2612f0f062136acaa9e1ed4c"
server_key_iv: "337812299c40116c297b317fcf4fa0ad"
server_key_version: "encryptionkey@0"
I20260619 16:46:19.005084 14115 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260619 16:46:19.008359 14130 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:46:19.009554 14115 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 16:46:19.009712 14115 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781887385903558-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781887385903558-388-0/minicluster-data/master-0/wal
uuid: "895a80903ac44b9ca0f30c9f12905bc5"
format_stamp: "Formatted at 2026-06-19 16:46:18 on dist-test-slave-wvz7"
server_key: "0b87e94c2612f0f062136acaa9e1ed4c"
server_key_iv: "337812299c40116c297b317fcf4fa0ad"
server_key_version: "encryptionkey@0"
I20260619 16:46:19.009892 14115 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781887385903558-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781887385903558-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781887385903558-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:46:19.040199 14115 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:46:19.044202 14115 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:46:19.044449 14115 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:46:19.061724 14115 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:38527
I20260619 16:46:19.061774 14181 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:38527 every 8 connection(s)
I20260619 16:46:19.063047 14115 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781887385903558-388-0/minicluster-data/master-0/data/info.pb
I20260619 16:46:19.066701 14182 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:46:19.069438 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu as pid 14115
I20260619 16:46:19.069650 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781887385903558-388-0/minicluster-data/master-0/wal/instance
I20260619 16:46:19.070463 388 external_mini_cluster.cc:1428] Setting key 21adc3660c38dada483940e083cbc766
I20260619 16:46:19.078778 14182 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 895a80903ac44b9ca0f30c9f12905bc5: Bootstrap starting.
I20260619 16:46:19.082427 14182 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 895a80903ac44b9ca0f30c9f12905bc5: Neither blocks nor log segments found. Creating new log.
I20260619 16:46:19.083665 14182 log.cc:824] T 00000000000000000000000000000000 P 895a80903ac44b9ca0f30c9f12905bc5: Log is configured to *not* fsync() on all Append() calls
Jun 19 16:46:19 dist-test-slave-wvz7 krb5kdc[14102](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887578, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 16:46:19.086764 14182 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 895a80903ac44b9ca0f30c9f12905bc5: No bootstrap required, opened a new log
I20260619 16:46:19.093384 14185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:19.072347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40578 (local address 127.0.97.62:38527)
0619 16:46:19.073056 (+ 709us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:19.073068 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:19.073108 (+ 40us) server_negotiation.cc:407] Connection header received
0619 16:46:19.074817 (+ 1709us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:19.074839 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:19.075233 (+ 394us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:19.075547 (+ 314us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:46:19.077247 (+ 1700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:19.078792 (+ 1545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:19.080655 (+ 1863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:19.081326 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:19.081465 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:19.086449 (+ 4984us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:46:19.086457 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:46:19.086471 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:46:19.086509 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:46:19.089428 (+ 2919us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:19.090274 (+ 846us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:19.090281 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:19.090287 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:19.090445 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:19.090790 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:19.090793 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:19.090795 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:19.091348 (+ 553us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:46:19.091571 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0619 16:46:19.091993 (+ 422us) server_negotiation.cc:299] Negotiation successful
0619 16:46:19.092364 (+ 371us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":520,"thread_start_us":317,"threads_started":1}
I20260619 16:46:19.095118 14182 raft_consensus.cc:348] T 00000000000000000000000000000000 P 895a80903ac44b9ca0f30c9f12905bc5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "895a80903ac44b9ca0f30c9f12905bc5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38527 } }
I20260619 16:46:19.095474 14182 raft_consensus.cc:374] T 00000000000000000000000000000000 P 895a80903ac44b9ca0f30c9f12905bc5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:46:19.095582 14182 raft_consensus.cc:729] T 00000000000000000000000000000000 P 895a80903ac44b9ca0f30c9f12905bc5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 895a80903ac44b9ca0f30c9f12905bc5, State: Initialized, Role: FOLLOWER
I20260619 16:46:19.096135 14182 consensus_queue.cc:260] T 00000000000000000000000000000000 P 895a80903ac44b9ca0f30c9f12905bc5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "895a80903ac44b9ca0f30c9f12905bc5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38527 } }
I20260619 16:46:19.096379 14182 raft_consensus.cc:388] T 00000000000000000000000000000000 P 895a80903ac44b9ca0f30c9f12905bc5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:46:19.096508 14182 raft_consensus.cc:482] T 00000000000000000000000000000000 P 895a80903ac44b9ca0f30c9f12905bc5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:46:19.096657 14182 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 895a80903ac44b9ca0f30c9f12905bc5 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:46:19.097095 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 16:46:19.098822 14182 raft_consensus.cc:504] T 00000000000000000000000000000000 P 895a80903ac44b9ca0f30c9f12905bc5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "895a80903ac44b9ca0f30c9f12905bc5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38527 } }
I20260619 16:46:19.099330 14182 leader_election.cc:302] T 00000000000000000000000000000000 P 895a80903ac44b9ca0f30c9f12905bc5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 895a80903ac44b9ca0f30c9f12905bc5; no voters:
I20260619 16:46:19.099864 14182 leader_election.cc:288] T 00000000000000000000000000000000 P 895a80903ac44b9ca0f30c9f12905bc5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 16:46:19.100011 14187 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 895a80903ac44b9ca0f30c9f12905bc5 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:46:19.100564 14187 raft_consensus.cc:686] T 00000000000000000000000000000000 P 895a80903ac44b9ca0f30c9f12905bc5 [term 1 LEADER]: Becoming Leader. State: Replica: 895a80903ac44b9ca0f30c9f12905bc5, State: Running, Role: LEADER
I20260619 16:46:19.101053 14187 consensus_queue.cc:237] T 00000000000000000000000000000000 P 895a80903ac44b9ca0f30c9f12905bc5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "895a80903ac44b9ca0f30c9f12905bc5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38527 } }
I20260619 16:46:19.102798 14188 sys_catalog.cc:455] T 00000000000000000000000000000000 P 895a80903ac44b9ca0f30c9f12905bc5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 895a80903ac44b9ca0f30c9f12905bc5. Latest consensus state: current_term: 1 leader_uuid: "895a80903ac44b9ca0f30c9f12905bc5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "895a80903ac44b9ca0f30c9f12905bc5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38527 } } }
I20260619 16:46:19.103235 14188 sys_catalog.cc:458] T 00000000000000000000000000000000 P 895a80903ac44b9ca0f30c9f12905bc5 [sys.catalog]: This master's current role is: LEADER
I20260619 16:46:19.103515 14182 sys_catalog.cc:564] T 00000000000000000000000000000000 P 895a80903ac44b9ca0f30c9f12905bc5 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:46:19.103495 14189 sys_catalog.cc:455] T 00000000000000000000000000000000 P 895a80903ac44b9ca0f30c9f12905bc5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "895a80903ac44b9ca0f30c9f12905bc5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "895a80903ac44b9ca0f30c9f12905bc5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38527 } } }
I20260619 16:46:19.103888 14189 sys_catalog.cc:458] T 00000000000000000000000000000000 P 895a80903ac44b9ca0f30c9f12905bc5 [sys.catalog]: This master's current role is: LEADER
I20260619 16:46:19.110039 14191 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:46:19.113977 14191 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:46:19.121520 14191 catalog_manager.cc:1347] Generated new cluster ID: a3f2942457654db89428c471821a2a84
I20260619 16:46:19.121635 14191 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:46:19.139380 14191 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 16:46:19.141218 14191 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:46:19.149668 14191 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 895a80903ac44b9ca0f30c9f12905bc5: Generated new TSK 0
I20260619 16:46:19.150735 14191 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 16:46:19.208597 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu pid 14115
I20260619 16:46:19.209781 14177 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:46:19.305243 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu with pid 14115
2026-06-19T16:46:19Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1857 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887385903558-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 16:46:19 dist-test-slave-wvz7 krb5kdc[14210](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 16:46:19 dist-test-slave-wvz7 krb5kdc[14210](info): set up 2 sockets
Jun 19 16:46:19 dist-test-slave-wvz7 krb5kdc[14210](info): commencing operation
krb5kdc: starting...
W20260619 16:46:20.301999 388 mini_kdc.cc:121] Time spent starting KDC: real 0.964s user 0.006s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887385903558-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887385903558-388-0/krb5kdc/test-user.keytab.
Jun 19 16:46:20 dist-test-slave-wvz7 krb5kdc[14210](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887580, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T16:46:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T16:46:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887385903558-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887385903558-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260619 16:46:20.500344 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu
/tmp/dist-test-taskEs2EH8/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887385903558-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887385903558-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887385903558-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887385903558-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45429
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887385903558-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38239
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:45429
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887385903558-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887385903558-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887385903558-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:46:20.805814 14223 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:46:20.806106 14223 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:46:20.806255 14223 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:46:20.813186 14223 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:46:20.813295 14223 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:46:20.813372 14223 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:46:20.813450 14223 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:46:20.813509 14223 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:46:20.822886 14223 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38239
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887385903558-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887385903558-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:45429
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887385903558-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45429
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887385903558-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887385903558-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:46:20.824074 14223 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:46:20.825296 14223 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:46:20.836009 14228 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:46:20.836066 14229 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:46:20.836071 14231 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:46:20.836954 14223 server_base.cc:1034] running on GCE node
I20260619 16:46:20.837826 14223 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:46:20.839325 14223 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:46:20.840555 14223 hybrid_clock.cc:648] HybridClock initialized: now 1781887580840506 us; error 39 us; skew 500 ppm
I20260619 16:46:20.840896 14223 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:46:20 dist-test-slave-wvz7 krb5kdc[14210](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887580, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:46:20.846801 14223 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260619 16:46:20.848232 14223 webserver.cc:460] Webserver started at http://127.0.97.62:34959/ using document root <none> and password file <none>
I20260619 16:46:20.848892 14223 fs_manager.cc:359] Metadata directory not provided
I20260619 16:46:20.849005 14223 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:46:20.849290 14223 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:46:20.852267 14223 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887385903558-388-0/minicluster-data/master-0/data/instance:
uuid: "e1e294d313564ec99776f3b6b38335d5"
format_stamp: "Formatted at 2026-06-19 16:46:20 on dist-test-slave-wvz7"
server_key: "fc92fef4748c44f2be23bd7ef93fe4f1"
server_key_iv: "529772319276df9c93bee0a3ed2a668d"
server_key_version: "encryptionkey@0"
I20260619 16:46:20.853005 14223 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887385903558-388-0/minicluster-data/master-0/wal/instance:
uuid: "e1e294d313564ec99776f3b6b38335d5"
format_stamp: "Formatted at 2026-06-19 16:46:20 on dist-test-slave-wvz7"
server_key: "fc92fef4748c44f2be23bd7ef93fe4f1"
server_key_iv: "529772319276df9c93bee0a3ed2a668d"
server_key_version: "encryptionkey@0"
I20260619 16:46:20.857980 14223 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260619 16:46:20.861200 14238 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:46:20.862550 14223 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 16:46:20.862700 14223 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887385903558-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887385903558-388-0/minicluster-data/master-0/wal
uuid: "e1e294d313564ec99776f3b6b38335d5"
format_stamp: "Formatted at 2026-06-19 16:46:20 on dist-test-slave-wvz7"
server_key: "fc92fef4748c44f2be23bd7ef93fe4f1"
server_key_iv: "529772319276df9c93bee0a3ed2a668d"
server_key_version: "encryptionkey@0"
I20260619 16:46:20.862866 14223 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887385903558-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887385903558-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887385903558-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:46:20.896113 14223 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:46:20.900064 14223 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:46:20.900326 14223 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:46:20.918210 14223 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:45429
I20260619 16:46:20.918268 14289 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:45429 every 8 connection(s)
I20260619 16:46:20.919498 14223 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887385903558-388-0/minicluster-data/master-0/data/info.pb
I20260619 16:46:20.923013 14290 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:46:20.929224 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu as pid 14223
I20260619 16:46:20.929438 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887385903558-388-0/minicluster-data/master-0/wal/instance
I20260619 16:46:20.930240 388 external_mini_cluster.cc:1428] Setting key d6b8d4de5ea66ed894099754d315cedb
I20260619 16:46:20.933560 14290 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e1e294d313564ec99776f3b6b38335d5: Bootstrap starting.
I20260619 16:46:20.937563 14290 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e1e294d313564ec99776f3b6b38335d5: Neither blocks nor log segments found. Creating new log.
I20260619 16:46:20.938714 14290 log.cc:824] T 00000000000000000000000000000000 P e1e294d313564ec99776f3b6b38335d5: Log is configured to *not* fsync() on all Append() calls
I20260619 16:46:20.941499 14290 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e1e294d313564ec99776f3b6b38335d5: No bootstrap required, opened a new log
Jun 19 16:46:20 dist-test-slave-wvz7 krb5kdc[14210](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887580, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 16:46:20.950340 14290 raft_consensus.cc:348] T 00000000000000000000000000000000 P e1e294d313564ec99776f3b6b38335d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1e294d313564ec99776f3b6b38335d5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45429 } }
I20260619 16:46:20.950732 14290 raft_consensus.cc:374] T 00000000000000000000000000000000 P e1e294d313564ec99776f3b6b38335d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:46:20.950830 14290 raft_consensus.cc:729] T 00000000000000000000000000000000 P e1e294d313564ec99776f3b6b38335d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1e294d313564ec99776f3b6b38335d5, State: Initialized, Role: FOLLOWER
I20260619 16:46:20.951364 14290 consensus_queue.cc:260] T 00000000000000000000000000000000 P e1e294d313564ec99776f3b6b38335d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1e294d313564ec99776f3b6b38335d5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45429 } }
I20260619 16:46:20.951583 14290 raft_consensus.cc:388] T 00000000000000000000000000000000 P e1e294d313564ec99776f3b6b38335d5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:46:20.951715 14290 raft_consensus.cc:482] T 00000000000000000000000000000000 P e1e294d313564ec99776f3b6b38335d5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:46:20.951879 14290 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e1e294d313564ec99776f3b6b38335d5 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:46:20.954190 14293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:20.932412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42932 (local address 127.0.97.62:45429)
0619 16:46:20.933323 (+ 911us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:20.933340 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:20.933690 (+ 350us) server_negotiation.cc:407] Connection header received
0619 16:46:20.936129 (+ 2439us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:20.936156 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:20.936650 (+ 494us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:20.937073 (+ 423us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:46:20.938383 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:20.940022 (+ 1639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:20.941287 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:20.942311 (+ 1024us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:20.942492 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:20.946222 (+ 3730us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:46:20.946232 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:46:20.946250 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:46:20.946306 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:46:20.950004 (+ 3698us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:20.950730 (+ 726us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:20.950738 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:20.950745 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:20.950948 (+ 203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:20.951351 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:20.951360 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:20.951364 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:20.951984 (+ 620us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:46:20.952267 (+ 283us) server_negotiation.cc:1035] Waiting for connection context
0619 16:46:20.952710 (+ 443us) server_negotiation.cc:299] Negotiation successful
0619 16:46:20.953116 (+ 406us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":648,"thread_start_us":396,"threads_started":1}
I20260619 16:46:20.954265 14290 raft_consensus.cc:504] T 00000000000000000000000000000000 P e1e294d313564ec99776f3b6b38335d5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1e294d313564ec99776f3b6b38335d5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45429 } }
I20260619 16:46:20.954792 14290 leader_election.cc:302] T 00000000000000000000000000000000 P e1e294d313564ec99776f3b6b38335d5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e1e294d313564ec99776f3b6b38335d5; no voters:
I20260619 16:46:20.955334 14290 leader_election.cc:288] T 00000000000000000000000000000000 P e1e294d313564ec99776f3b6b38335d5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 16:46:20.955497 14295 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e1e294d313564ec99776f3b6b38335d5 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:46:20.957453 14295 raft_consensus.cc:686] T 00000000000000000000000000000000 P e1e294d313564ec99776f3b6b38335d5 [term 1 LEADER]: Becoming Leader. State: Replica: e1e294d313564ec99776f3b6b38335d5, State: Running, Role: LEADER
I20260619 16:46:20.957458 14290 sys_catalog.cc:564] T 00000000000000000000000000000000 P e1e294d313564ec99776f3b6b38335d5 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:46:20.958067 14295 consensus_queue.cc:237] T 00000000000000000000000000000000 P e1e294d313564ec99776f3b6b38335d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1e294d313564ec99776f3b6b38335d5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45429 } }
I20260619 16:46:20.960536 14297 sys_catalog.cc:455] T 00000000000000000000000000000000 P e1e294d313564ec99776f3b6b38335d5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e1e294d313564ec99776f3b6b38335d5. Latest consensus state: current_term: 1 leader_uuid: "e1e294d313564ec99776f3b6b38335d5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1e294d313564ec99776f3b6b38335d5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45429 } } }
I20260619 16:46:20.960870 14297 sys_catalog.cc:458] T 00000000000000000000000000000000 P e1e294d313564ec99776f3b6b38335d5 [sys.catalog]: This master's current role is: LEADER
I20260619 16:46:20.961196 14296 sys_catalog.cc:455] T 00000000000000000000000000000000 P e1e294d313564ec99776f3b6b38335d5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e1e294d313564ec99776f3b6b38335d5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1e294d313564ec99776f3b6b38335d5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45429 } } }
I20260619 16:46:20.961513 14296 sys_catalog.cc:458] T 00000000000000000000000000000000 P e1e294d313564ec99776f3b6b38335d5 [sys.catalog]: This master's current role is: LEADER
I20260619 16:46:20.962927 14303 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:46:20.967999 14303 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:46:20.977063 14303 catalog_manager.cc:1347] Generated new cluster ID: b3db8dcb2ff3459f9a5b16ab9e88649d
I20260619 16:46:20.977159 14303 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:46:20.986425 14303 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 16:46:20.988200 14303 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:46:21.000563 14303 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e1e294d313564ec99776f3b6b38335d5: Generated new TSK 0
I20260619 16:46:21.001583 14303 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887385903558-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887385903558-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260619 16:46:21.058145 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu
/tmp/dist-test-taskEs2EH8/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887385903558-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887385903558-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887385903558-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887385903558-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45429
--keytab_file=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887385903558-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38239
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887385903558-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887385903558-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887385903558-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:46:21.368090 14316 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:46:21.368358 14316 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:46:21.368492 14316 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:46:21.375489 14316 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:46:21.375603 14316 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:46:21.375751 14316 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260619 16:46:21.384915 14316 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38239
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887385903558-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887385903558-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887385903558-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887385903558-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45429
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887385903558-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:46:21.386134 14316 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:46:21.387342 14316 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:46:21.399473 14321 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:46:21.399411 14322 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:46:21.399439 14324 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:46:21.400333 14316 server_base.cc:1034] running on GCE node
I20260619 16:46:21.400877 14316 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:46:21.401714 14316 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:46:21.402940 14316 hybrid_clock.cc:648] HybridClock initialized: now 1781887581402899 us; error 39 us; skew 500 ppm
I20260619 16:46:21.403228 14316 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:46:21 dist-test-slave-wvz7 krb5kdc[14210](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887581, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:46:21.406172 14316 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260619 16:46:21.407644 14316 webserver.cc:460] Webserver started at http://127.0.97.1:42735/ using document root <none> and password file <none>
I20260619 16:46:21.408274 14316 fs_manager.cc:359] Metadata directory not provided
I20260619 16:46:21.408385 14316 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:46:21.408648 14316 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:46:21.411496 14316 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887385903558-388-0/minicluster-data/ts-0/data/instance:
uuid: "a9dc2ad605434cb49fac1258d87806f5"
format_stamp: "Formatted at 2026-06-19 16:46:21 on dist-test-slave-wvz7"
server_key: "a1fdf62f258763c823e087b9c5d87f23"
server_key_iv: "8a28126bc939a78919f716c3739ca8da"
server_key_version: "encryptionkey@0"
I20260619 16:46:21.412220 14316 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887385903558-388-0/minicluster-data/ts-0/wal/instance:
uuid: "a9dc2ad605434cb49fac1258d87806f5"
format_stamp: "Formatted at 2026-06-19 16:46:21 on dist-test-slave-wvz7"
server_key: "a1fdf62f258763c823e087b9c5d87f23"
server_key_iv: "8a28126bc939a78919f716c3739ca8da"
server_key_version: "encryptionkey@0"
I20260619 16:46:21.417101 14316 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260619 16:46:21.420436 14331 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:46:21.421687 14316 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 16:46:21.421871 14316 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887385903558-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887385903558-388-0/minicluster-data/ts-0/wal
uuid: "a9dc2ad605434cb49fac1258d87806f5"
format_stamp: "Formatted at 2026-06-19 16:46:21 on dist-test-slave-wvz7"
server_key: "a1fdf62f258763c823e087b9c5d87f23"
server_key_iv: "8a28126bc939a78919f716c3739ca8da"
server_key_version: "encryptionkey@0"
I20260619 16:46:21.422039 14316 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887385903558-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887385903558-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887385903558-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:46:21.453431 14316 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:46:21.457700 14316 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:46:21.458060 14316 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:46:21.459022 14316 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:46:21.460286 14316 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:46:21.460356 14316 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:46:21.460446 14316 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:46:21.460487 14316 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:46:21.488929 14316 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:37367
I20260619 16:46:21.488974 14443 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:37367 every 8 connection(s)
I20260619 16:46:21.490171 14316 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887385903558-388-0/minicluster-data/ts-0/data/info.pb
I20260619 16:46:21.490980 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu as pid 14316
I20260619 16:46:21.491209 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887385903558-388-0/minicluster-data/ts-0/wal/instance
I20260619 16:46:21.491983 388 external_mini_cluster.cc:1428] Setting key 8bd7dc050fad49e209caad93eff25509
Jun 19 16:46:21 dist-test-slave-wvz7 krb5kdc[14210](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887581, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 16:46:21.510375 14446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:21.493092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:33185 (local address 127.0.97.62:45429)
0619 16:46:21.493502 (+ 410us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:21.493507 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:21.494594 (+ 1087us) server_negotiation.cc:407] Connection header received
0619 16:46:21.496334 (+ 1740us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:21.496340 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:21.496428 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:21.496518 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:46:21.498007 (+ 1489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:21.498747 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:21.500300 (+ 1553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:21.500949 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:21.501072 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:21.504659 (+ 3587us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:46:21.504673 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:46:21.504678 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:46:21.504714 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:46:21.507532 (+ 2818us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:21.508170 (+ 638us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:21.508175 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:21.508179 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:21.508312 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:21.508688 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:21.508692 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:21.508695 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:21.509019 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:46:21.509159 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0619 16:46:21.509822 (+ 663us) server_negotiation.cc:299] Negotiation successful
0619 16:46:21.510031 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":311,"thread_start_us":219,"threads_started":1}
I20260619 16:46:21.511261 14447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:21.493784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45429 (local address 127.0.97.1:33185)
0619 16:46:21.494460 (+ 676us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:21.494497 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0619 16:46:21.496087 (+ 1590us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:46:21.496758 (+ 671us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:46:21.496766 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:46:21.497262 (+ 496us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:46:21.497854 (+ 592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:21.497874 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:21.498887 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:21.498891 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:21.500109 (+ 1218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:21.500124 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:21.501163 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:21.501170 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:21.501430 (+ 260us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:21.502089 (+ 659us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:46:21.502116 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:46:21.504450 (+ 2334us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:46:21.507668 (+ 3218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:46:21.507674 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:46:21.507685 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:46:21.508061 (+ 376us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:46:21.508408 (+ 347us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:46:21.508412 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:46:21.508414 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:46:21.508580 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:46:21.509157 (+ 577us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:46:21.509163 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:46:21.509452 (+ 289us) client_negotiation.cc:769] Sending connection context
0619 16:46:21.509671 (+ 219us) client_negotiation.cc:240] Negotiation successful
0619 16:46:21.510187 (+ 516us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":443,"thread_start_us":280,"threads_started":1}
I20260619 16:46:21.512714 14444 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45429
I20260619 16:46:21.513021 14444 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260619 16:46:21.513828 14444 heartbeater.cc:511] Master 127.0.97.62:45429 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260619 16:46:21.515800 14255 ts_manager.cc:194] Registered new tserver with Master: a9dc2ad605434cb49fac1258d87806f5 (127.0.97.1:37367)
I20260619 16:46:21.517200 14255 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:33185
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887385903558-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887385903558-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260619 16:46:21.538407 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu
/tmp/dist-test-taskEs2EH8/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887385903558-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887385903558-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887385903558-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887385903558-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45429
--keytab_file=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887385903558-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38239
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887385903558-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887385903558-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887385903558-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:46:21.838411 14450 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:46:21.838686 14450 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:46:21.838806 14450 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:46:21.845920 14450 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:46:21.846025 14450 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:46:21.846179 14450 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260619 16:46:21.855712 14450 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38239
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887385903558-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887385903558-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887385903558-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887385903558-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45429
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887385903558-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:46:21.856976 14450 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:46:21.858291 14450 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:46:21.871093 14455 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:46:21.871115 14456 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:46:21.871142 14458 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:46:21.871827 14450 server_base.cc:1034] running on GCE node
I20260619 16:46:21.872404 14450 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:46:21.873299 14450 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:46:21.874512 14450 hybrid_clock.cc:648] HybridClock initialized: now 1781887581874465 us; error 34 us; skew 500 ppm
I20260619 16:46:21.874820 14450 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:46:21 dist-test-slave-wvz7 krb5kdc[14210](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887581, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:46:21.877918 14450 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260619 16:46:21.879400 14450 webserver.cc:460] Webserver started at http://127.0.97.2:35761/ using document root <none> and password file <none>
I20260619 16:46:21.880053 14450 fs_manager.cc:359] Metadata directory not provided
I20260619 16:46:21.880151 14450 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:46:21.880425 14450 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:46:21.883286 14450 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887385903558-388-0/minicluster-data/ts-1/data/instance:
uuid: "09f7fa28cb44464fa809ee1bcf1a2c03"
format_stamp: "Formatted at 2026-06-19 16:46:21 on dist-test-slave-wvz7"
server_key: "f18fa1c04ddfbf586c6a8778a22869b0"
server_key_iv: "7dfdcd84e637eff4f21842baa7227bf2"
server_key_version: "encryptionkey@0"
I20260619 16:46:21.884028 14450 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887385903558-388-0/minicluster-data/ts-1/wal/instance:
uuid: "09f7fa28cb44464fa809ee1bcf1a2c03"
format_stamp: "Formatted at 2026-06-19 16:46:21 on dist-test-slave-wvz7"
server_key: "f18fa1c04ddfbf586c6a8778a22869b0"
server_key_iv: "7dfdcd84e637eff4f21842baa7227bf2"
server_key_version: "encryptionkey@0"
I20260619 16:46:21.891361 14450 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.007s sys 0.001s
I20260619 16:46:21.895295 14465 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:46:21.896659 14450 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 16:46:21.896800 14450 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887385903558-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887385903558-388-0/minicluster-data/ts-1/wal
uuid: "09f7fa28cb44464fa809ee1bcf1a2c03"
format_stamp: "Formatted at 2026-06-19 16:46:21 on dist-test-slave-wvz7"
server_key: "f18fa1c04ddfbf586c6a8778a22869b0"
server_key_iv: "7dfdcd84e637eff4f21842baa7227bf2"
server_key_version: "encryptionkey@0"
I20260619 16:46:21.896991 14450 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887385903558-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887385903558-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887385903558-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:46:21.924494 14450 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:46:21.928632 14450 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:46:21.928891 14450 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:46:21.929831 14450 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:46:21.931190 14450 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:46:21.931262 14450 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:46:21.931370 14450 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:46:21.931420 14450 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:46:21.960338 14450 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:42613
I20260619 16:46:21.960377 14577 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:42613 every 8 connection(s)
I20260619 16:46:21.961493 14450 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887385903558-388-0/minicluster-data/ts-1/data/info.pb
I20260619 16:46:21.967028 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu as pid 14450
I20260619 16:46:21.967212 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887385903558-388-0/minicluster-data/ts-1/wal/instance
I20260619 16:46:21.967628 388 external_mini_cluster.cc:1428] Setting key dba58bea67f595724640ad528802439a
Jun 19 16:46:21 dist-test-slave-wvz7 krb5kdc[14210](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887581, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 16:46:21.980579 14446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:21.963960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:48405 (local address 127.0.97.62:45429)
0619 16:46:21.964121 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:21.964125 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:21.965297 (+ 1172us) server_negotiation.cc:407] Connection header received
0619 16:46:21.967202 (+ 1905us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:21.967208 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:21.967298 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:21.967394 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:46:21.968937 (+ 1543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:21.969825 (+ 888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:21.971320 (+ 1495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:21.971713 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:21.971821 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:21.975017 (+ 3196us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:46:21.975022 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:46:21.975026 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:46:21.975057 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:46:21.977721 (+ 2664us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:21.978413 (+ 692us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:21.978419 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:21.978423 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:21.978563 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:21.978945 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:21.978950 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:21.978954 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:21.979231 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:46:21.979361 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0619 16:46:21.980008 (+ 647us) server_negotiation.cc:299] Negotiation successful
0619 16:46:21.980247 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260619 16:46:21.981498 14580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:21.964488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45429 (local address 127.0.97.2:48405)
0619 16:46:21.965140 (+ 652us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:21.965178 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0619 16:46:21.966903 (+ 1725us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:46:21.967623 (+ 720us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:46:21.967631 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:46:21.968138 (+ 507us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:46:21.968783 (+ 645us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:21.968802 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:21.969973 (+ 1171us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:21.969977 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:21.971182 (+ 1205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:21.971198 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:21.971910 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:21.971917 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:21.972127 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:21.972735 (+ 608us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:46:21.972757 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:46:21.974819 (+ 2062us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:46:21.977871 (+ 3052us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:46:21.977880 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:46:21.977897 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:46:21.978302 (+ 405us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:46:21.978668 (+ 366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:46:21.978673 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:46:21.978675 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:46:21.978842 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:46:21.979346 (+ 504us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:46:21.979354 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:46:21.979665 (+ 311us) client_negotiation.cc:769] Sending connection context
0619 16:46:21.979906 (+ 241us) client_negotiation.cc:240] Negotiation successful
0619 16:46:21.980407 (+ 501us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":443,"thread_start_us":256,"threads_started":1}
I20260619 16:46:21.982976 14578 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45429
I20260619 16:46:21.983310 14578 heartbeater.cc:464] Registering TS with master...
I20260619 16:46:21.984148 14578 heartbeater.cc:511] Master 127.0.97.62:45429 requested a full tablet report, sending...
I20260619 16:46:21.985877 14255 ts_manager.cc:194] Registered new tserver with Master: 09f7fa28cb44464fa809ee1bcf1a2c03 (127.0.97.2:42613)
I20260619 16:46:21.986963 14255 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:48405
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887385903558-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887385903558-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260619 16:46:22.013599 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu
/tmp/dist-test-taskEs2EH8/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887385903558-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887385903558-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887385903558-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887385903558-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45429
--keytab_file=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887385903558-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38239
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887385903558-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887385903558-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887385903558-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:46:22.320926 14583 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:46:22.321206 14583 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:46:22.321297 14583 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:46:22.328626 14583 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:46:22.328744 14583 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:46:22.328860 14583 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260619 16:46:22.338465 14583 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38239
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887385903558-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887385903558-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887385903558-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887385903558-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45429
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887385903558-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:46:22.339679 14583 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:46:22.340915 14583 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:46:22.353561 14588 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:46:22.353677 14589 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:46:22.353654 14591 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:46:22.354539 14583 server_base.cc:1034] running on GCE node
I20260619 16:46:22.355106 14583 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:46:22.356009 14583 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:46:22.357224 14583 hybrid_clock.cc:648] HybridClock initialized: now 1781887582357177 us; error 40 us; skew 500 ppm
I20260619 16:46:22.357543 14583 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:46:22 dist-test-slave-wvz7 krb5kdc[14210](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887582, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:46:22.360549 14583 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260619 16:46:22.362012 14583 webserver.cc:460] Webserver started at http://127.0.97.3:42761/ using document root <none> and password file <none>
I20260619 16:46:22.362666 14583 fs_manager.cc:359] Metadata directory not provided
I20260619 16:46:22.362782 14583 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:46:22.363056 14583 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:46:22.365943 14583 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887385903558-388-0/minicluster-data/ts-2/data/instance:
uuid: "f2689ba3f5fb4c70bafaa2d05232705a"
format_stamp: "Formatted at 2026-06-19 16:46:22 on dist-test-slave-wvz7"
server_key: "f8f2beb316fac04fe57535d8da6ec7d5"
server_key_iv: "1142e95cb44a1e97d33f4af87f3b0bf4"
server_key_version: "encryptionkey@0"
I20260619 16:46:22.366700 14583 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887385903558-388-0/minicluster-data/ts-2/wal/instance:
uuid: "f2689ba3f5fb4c70bafaa2d05232705a"
format_stamp: "Formatted at 2026-06-19 16:46:22 on dist-test-slave-wvz7"
server_key: "f8f2beb316fac04fe57535d8da6ec7d5"
server_key_iv: "1142e95cb44a1e97d33f4af87f3b0bf4"
server_key_version: "encryptionkey@0"
I20260619 16:46:22.371656 14583 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 16:46:22.375056 14598 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:46:22.376293 14583 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260619 16:46:22.376447 14583 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887385903558-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887385903558-388-0/minicluster-data/ts-2/wal
uuid: "f2689ba3f5fb4c70bafaa2d05232705a"
format_stamp: "Formatted at 2026-06-19 16:46:22 on dist-test-slave-wvz7"
server_key: "f8f2beb316fac04fe57535d8da6ec7d5"
server_key_iv: "1142e95cb44a1e97d33f4af87f3b0bf4"
server_key_version: "encryptionkey@0"
I20260619 16:46:22.376621 14583 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887385903558-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887385903558-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887385903558-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:46:22.399856 14583 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:46:22.404069 14583 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:46:22.404340 14583 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:46:22.405256 14583 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:46:22.406558 14583 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:46:22.406633 14583 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:46:22.406747 14583 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:46:22.406798 14583 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:46:22.436829 14583 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:37273
I20260619 16:46:22.436862 14710 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:37273 every 8 connection(s)
I20260619 16:46:22.438163 14583 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887385903558-388-0/minicluster-data/ts-2/data/info.pb
I20260619 16:46:22.443884 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu as pid 14583
I20260619 16:46:22.444067 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887385903558-388-0/minicluster-data/ts-2/wal/instance
I20260619 16:46:22.444581 388 external_mini_cluster.cc:1428] Setting key d2d894993cd0ea65cf5f1ff2f044edff
Jun 19 16:46:22 dist-test-slave-wvz7 krb5kdc[14210](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887582, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 16:46:22.457041 14446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:22.440754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:57511 (local address 127.0.97.62:45429)
0619 16:46:22.440942 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:22.440947 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:22.442068 (+ 1121us) server_negotiation.cc:407] Connection header received
0619 16:46:22.443968 (+ 1900us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:22.443975 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:22.444056 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:22.444155 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:46:22.445853 (+ 1698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:22.446767 (+ 914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:22.448324 (+ 1557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:22.448695 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:22.448785 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:22.451935 (+ 3150us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:46:22.451939 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:46:22.451942 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:46:22.451976 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:46:22.454480 (+ 2504us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:22.455120 (+ 640us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:22.455124 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:22.455126 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:22.455233 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:22.455641 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:22.455644 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:22.455646 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:22.455903 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:46:22.456007 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0619 16:46:22.456632 (+ 625us) server_negotiation.cc:299] Negotiation successful
0619 16:46:22.456786 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260619 16:46:22.458088 14713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:22.441218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45429 (local address 127.0.97.3:57511)
0619 16:46:22.441907 (+ 689us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:22.441957 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0619 16:46:22.443649 (+ 1692us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:46:22.444409 (+ 760us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:46:22.444420 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:46:22.445120 (+ 700us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:46:22.445677 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:22.445694 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:22.446900 (+ 1206us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:22.446904 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:22.448202 (+ 1298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:22.448214 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:22.448857 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:22.448861 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:22.449046 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:22.449586 (+ 540us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:46:22.449606 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:46:22.451675 (+ 2069us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:46:22.454618 (+ 2943us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:46:22.454624 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:46:22.454635 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:46:22.455029 (+ 394us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:46:22.455350 (+ 321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:46:22.455354 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:46:22.455356 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:46:22.455551 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:46:22.456024 (+ 473us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:46:22.456030 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:46:22.456337 (+ 307us) client_negotiation.cc:769] Sending connection context
0619 16:46:22.456554 (+ 217us) client_negotiation.cc:240] Negotiation successful
0619 16:46:22.456932 (+ 378us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":439,"thread_start_us":249,"threads_started":1}
I20260619 16:46:22.459373 14711 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45429
I20260619 16:46:22.459662 14711 heartbeater.cc:464] Registering TS with master...
I20260619 16:46:22.460421 14711 heartbeater.cc:511] Master 127.0.97.62:45429 requested a full tablet report, sending...
I20260619 16:46:22.461681 14255 ts_manager.cc:194] Registered new tserver with Master: f2689ba3f5fb4c70bafaa2d05232705a (127.0.97.3:37273)
I20260619 16:46:22.462471 14255 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:57511
I20260619 16:46:22.471691 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 16:46:22.487444 14446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:22.475734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42940 (local address 127.0.97.62:45429)
0619 16:46:22.475892 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:22.475896 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:22.476291 (+ 395us) server_negotiation.cc:407] Connection header received
0619 16:46:22.476438 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:22.476442 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:22.476503 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:22.476572 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:46:22.477839 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:22.478533 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:22.479893 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:22.480344 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:22.480433 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:22.482620 (+ 2187us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:46:22.482624 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:46:22.482628 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:46:22.482658 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:46:22.485074 (+ 2416us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:22.485659 (+ 585us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:22.485664 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:22.485666 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:22.485819 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:22.486184 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:22.486188 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:22.486190 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:22.486518 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:46:22.486700 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0619 16:46:22.487021 (+ 321us) server_negotiation.cc:299] Negotiation successful
0619 16:46:22.487195 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260619 16:46:22.491384 14255 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42940:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 16:46:22.498208 14255 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 16:46:22.519959 14444 heartbeater.cc:503] Master 127.0.97.62:45429 was elected leader, sending a full tablet report...
I20260619 16:46:22.520242 14723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:22.509679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:37367 (local address 127.0.0.1:47342)
0619 16:46:22.510486 (+ 807us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:22.510525 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0619 16:46:22.510684 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:46:22.511912 (+ 1228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:46:22.511921 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:46:22.511955 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:46:22.512388 (+ 433us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:22.512400 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:22.514692 (+ 2292us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:22.514698 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:22.517168 (+ 2470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:22.517185 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:22.519238 (+ 2053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:22.519245 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:22.519455 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:22.519510 (+ 55us) client_negotiation.cc:769] Sending connection context
0619 16:46:22.519619 (+ 109us) client_negotiation.cc:240] Negotiation successful
0619 16:46:22.519819 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":574,"thread_start_us":440,"threads_started":1}
I20260619 16:46:22.521692 14725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:22.509972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47342 (local address 127.0.97.1:37367)
0619 16:46:22.511247 (+ 1275us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:22.511257 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:22.511288 (+ 31us) server_negotiation.cc:407] Connection header received
0619 16:46:22.511390 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:22.511398 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:22.511655 (+ 257us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:22.512610 (+ 955us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:46:22.512864 (+ 254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:22.514520 (+ 1656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:22.517401 (+ 2881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:22.518994 (+ 1593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:22.520849 (+ 1855us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:22.521108 (+ 259us) server_negotiation.cc:1035] Waiting for connection context
0619 16:46:22.521175 (+ 67us) server_negotiation.cc:299] Negotiation successful
0619 16:46:22.521348 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1104,"thread_start_us":1567,"threads_started":1}
I20260619 16:46:22.526098 14379 tablet_service.cc:1467] Processing CreateTablet for tablet d73d296839b04eedacbdb05c96d206a1 (DEFAULT_TABLE table=test-table [id=f3786b1d03274b5c9ce59a9404170428]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:46:22.527513 14379 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d73d296839b04eedacbdb05c96d206a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:46:22.531828 14727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:22.511388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:42613 (local address 127.0.0.1:52230)
0619 16:46:22.521399 (+ 10011us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:22.521417 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 16:46:22.521528 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:46:22.524025 (+ 2497us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:46:22.524029 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:46:22.524052 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:46:22.524391 (+ 339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:22.524403 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:22.526592 (+ 2189us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:22.526597 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:22.528660 (+ 2063us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:22.528673 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:22.531069 (+ 2396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:22.531075 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:22.531246 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:22.531281 (+ 35us) client_negotiation.cc:769] Sending connection context
0619 16:46:22.531347 (+ 66us) client_negotiation.cc:240] Negotiation successful
0619 16:46:22.531478 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9887,"thread_start_us":10210,"threads_started":1}
I20260619 16:46:22.532917 14728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:22.511509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52230 (local address 127.0.97.2:42613)
0619 16:46:22.523431 (+ 11922us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:22.523439 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:22.523467 (+ 28us) server_negotiation.cc:407] Connection header received
0619 16:46:22.523565 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:22.523572 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:22.523824 (+ 252us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:22.524601 (+ 777us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:46:22.524850 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:22.526459 (+ 1609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:22.529879 (+ 3420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:22.530866 (+ 987us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:22.532192 (+ 1326us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:22.532381 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0619 16:46:22.532454 (+ 73us) server_negotiation.cc:299] Negotiation successful
0619 16:46:22.532597 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11768,"thread_start_us":13237,"threads_started":1}
I20260619 16:46:22.532970 14723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:22.509848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:37273 (local address 127.0.0.1:59096)
0619 16:46:22.520553 (+ 10705us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:22.520571 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 16:46:22.520684 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:46:22.526603 (+ 5919us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:46:22.526607 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:46:22.526634 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:46:22.526878 (+ 244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:22.526890 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:22.528814 (+ 1924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:22.528819 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:22.530550 (+ 1731us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:22.530574 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:22.532246 (+ 1672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:22.532252 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:22.532430 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:22.532473 (+ 43us) client_negotiation.cc:769] Sending connection context
0619 16:46:22.532564 (+ 91us) client_negotiation.cc:240] Negotiation successful
0619 16:46:22.532701 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10615,"thread_start_us":13137,"threads_started":1}
I20260619 16:46:22.533797 14726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:22.510923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59096 (local address 127.0.97.3:37273)
0619 16:46:22.526000 (+ 15077us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:22.526008 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:22.526041 (+ 33us) server_negotiation.cc:407] Connection header received
0619 16:46:22.526125 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:22.526132 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:22.526349 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:22.526601 (+ 252us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:46:22.527057 (+ 456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:22.528660 (+ 1603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:22.530725 (+ 2065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:22.532096 (+ 1371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:22.532215 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:22.532414 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0619 16:46:22.532606 (+ 192us) server_negotiation.cc:299] Negotiation successful
0619 16:46:22.532755 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14908,"thread_start_us":14735,"threads_started":1}
I20260619 16:46:22.536580 14513 tablet_service.cc:1467] Processing CreateTablet for tablet d73d296839b04eedacbdb05c96d206a1 (DEFAULT_TABLE table=test-table [id=f3786b1d03274b5c9ce59a9404170428]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:46:22.536813 14646 tablet_service.cc:1467] Processing CreateTablet for tablet d73d296839b04eedacbdb05c96d206a1 (DEFAULT_TABLE table=test-table [id=f3786b1d03274b5c9ce59a9404170428]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:46:22.537945 14646 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d73d296839b04eedacbdb05c96d206a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:46:22.537985 14513 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d73d296839b04eedacbdb05c96d206a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:46:22.541967 14729 tablet_bootstrap.cc:492] T d73d296839b04eedacbdb05c96d206a1 P a9dc2ad605434cb49fac1258d87806f5: Bootstrap starting.
I20260619 16:46:22.545442 14729 tablet_bootstrap.cc:654] T d73d296839b04eedacbdb05c96d206a1 P a9dc2ad605434cb49fac1258d87806f5: Neither blocks nor log segments found. Creating new log.
I20260619 16:46:22.547027 14729 log.cc:824] T d73d296839b04eedacbdb05c96d206a1 P a9dc2ad605434cb49fac1258d87806f5: Log is configured to *not* fsync() on all Append() calls
I20260619 16:46:22.550300 14731 tablet_bootstrap.cc:492] T d73d296839b04eedacbdb05c96d206a1 P 09f7fa28cb44464fa809ee1bcf1a2c03: Bootstrap starting.
I20260619 16:46:22.551213 14729 tablet_bootstrap.cc:492] T d73d296839b04eedacbdb05c96d206a1 P a9dc2ad605434cb49fac1258d87806f5: No bootstrap required, opened a new log
I20260619 16:46:22.551512 14729 ts_tablet_manager.cc:1397] T d73d296839b04eedacbdb05c96d206a1 P a9dc2ad605434cb49fac1258d87806f5: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260619 16:46:22.553246 14731 tablet_bootstrap.cc:654] T d73d296839b04eedacbdb05c96d206a1 P 09f7fa28cb44464fa809ee1bcf1a2c03: Neither blocks nor log segments found. Creating new log.
I20260619 16:46:22.554437 14731 log.cc:824] T d73d296839b04eedacbdb05c96d206a1 P 09f7fa28cb44464fa809ee1bcf1a2c03: Log is configured to *not* fsync() on all Append() calls
I20260619 16:46:22.556408 14732 tablet_bootstrap.cc:492] T d73d296839b04eedacbdb05c96d206a1 P f2689ba3f5fb4c70bafaa2d05232705a: Bootstrap starting.
I20260619 16:46:22.556995 14731 tablet_bootstrap.cc:492] T d73d296839b04eedacbdb05c96d206a1 P 09f7fa28cb44464fa809ee1bcf1a2c03: No bootstrap required, opened a new log
I20260619 16:46:22.557215 14731 ts_tablet_manager.cc:1397] T d73d296839b04eedacbdb05c96d206a1 P 09f7fa28cb44464fa809ee1bcf1a2c03: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.003s
I20260619 16:46:22.559877 14732 tablet_bootstrap.cc:654] T d73d296839b04eedacbdb05c96d206a1 P f2689ba3f5fb4c70bafaa2d05232705a: Neither blocks nor log segments found. Creating new log.
I20260619 16:46:22.561033 14732 log.cc:824] T d73d296839b04eedacbdb05c96d206a1 P f2689ba3f5fb4c70bafaa2d05232705a: Log is configured to *not* fsync() on all Append() calls
I20260619 16:46:22.561862 14729 raft_consensus.cc:348] T d73d296839b04eedacbdb05c96d206a1 P a9dc2ad605434cb49fac1258d87806f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9dc2ad605434cb49fac1258d87806f5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37367 } } peers { permanent_uuid: "f2689ba3f5fb4c70bafaa2d05232705a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37273 } } peers { permanent_uuid: "09f7fa28cb44464fa809ee1bcf1a2c03" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42613 } }
I20260619 16:46:22.562321 14729 raft_consensus.cc:374] T d73d296839b04eedacbdb05c96d206a1 P a9dc2ad605434cb49fac1258d87806f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:46:22.562422 14729 raft_consensus.cc:729] T d73d296839b04eedacbdb05c96d206a1 P a9dc2ad605434cb49fac1258d87806f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a9dc2ad605434cb49fac1258d87806f5, State: Initialized, Role: FOLLOWER
I20260619 16:46:22.563102 14729 consensus_queue.cc:260] T d73d296839b04eedacbdb05c96d206a1 P a9dc2ad605434cb49fac1258d87806f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9dc2ad605434cb49fac1258d87806f5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37367 } } peers { permanent_uuid: "f2689ba3f5fb4c70bafaa2d05232705a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37273 } } peers { permanent_uuid: "09f7fa28cb44464fa809ee1bcf1a2c03" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42613 } }
I20260619 16:46:22.564653 14732 tablet_bootstrap.cc:492] T d73d296839b04eedacbdb05c96d206a1 P f2689ba3f5fb4c70bafaa2d05232705a: No bootstrap required, opened a new log
I20260619 16:46:22.564752 14729 ts_tablet_manager.cc:1428] T d73d296839b04eedacbdb05c96d206a1 P a9dc2ad605434cb49fac1258d87806f5: Time spent starting tablet: real 0.013s user 0.008s sys 0.004s
I20260619 16:46:22.564960 14732 ts_tablet_manager.cc:1397] T d73d296839b04eedacbdb05c96d206a1 P f2689ba3f5fb4c70bafaa2d05232705a: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.007s
I20260619 16:46:22.564930 14731 raft_consensus.cc:348] T d73d296839b04eedacbdb05c96d206a1 P 09f7fa28cb44464fa809ee1bcf1a2c03 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9dc2ad605434cb49fac1258d87806f5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37367 } } peers { permanent_uuid: "f2689ba3f5fb4c70bafaa2d05232705a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37273 } } peers { permanent_uuid: "09f7fa28cb44464fa809ee1bcf1a2c03" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42613 } }
I20260619 16:46:22.565349 14731 raft_consensus.cc:374] T d73d296839b04eedacbdb05c96d206a1 P 09f7fa28cb44464fa809ee1bcf1a2c03 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:46:22.565448 14731 raft_consensus.cc:729] T d73d296839b04eedacbdb05c96d206a1 P 09f7fa28cb44464fa809ee1bcf1a2c03 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09f7fa28cb44464fa809ee1bcf1a2c03, State: Initialized, Role: FOLLOWER
I20260619 16:46:22.566069 14731 consensus_queue.cc:260] T d73d296839b04eedacbdb05c96d206a1 P 09f7fa28cb44464fa809ee1bcf1a2c03 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9dc2ad605434cb49fac1258d87806f5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37367 } } peers { permanent_uuid: "f2689ba3f5fb4c70bafaa2d05232705a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37273 } } peers { permanent_uuid: "09f7fa28cb44464fa809ee1bcf1a2c03" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42613 } }
I20260619 16:46:22.567248 14578 heartbeater.cc:503] Master 127.0.97.62:45429 was elected leader, sending a full tablet report...
I20260619 16:46:22.567824 14731 ts_tablet_manager.cc:1428] T d73d296839b04eedacbdb05c96d206a1 P 09f7fa28cb44464fa809ee1bcf1a2c03: Time spent starting tablet: real 0.011s user 0.010s sys 0.000s
I20260619 16:46:22.573451 14732 raft_consensus.cc:348] T d73d296839b04eedacbdb05c96d206a1 P f2689ba3f5fb4c70bafaa2d05232705a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9dc2ad605434cb49fac1258d87806f5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37367 } } peers { permanent_uuid: "f2689ba3f5fb4c70bafaa2d05232705a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37273 } } peers { permanent_uuid: "09f7fa28cb44464fa809ee1bcf1a2c03" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42613 } }
I20260619 16:46:22.573783 14732 raft_consensus.cc:374] T d73d296839b04eedacbdb05c96d206a1 P f2689ba3f5fb4c70bafaa2d05232705a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:46:22.573864 14732 raft_consensus.cc:729] T d73d296839b04eedacbdb05c96d206a1 P f2689ba3f5fb4c70bafaa2d05232705a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f2689ba3f5fb4c70bafaa2d05232705a, State: Initialized, Role: FOLLOWER
I20260619 16:46:22.574376 14732 consensus_queue.cc:260] T d73d296839b04eedacbdb05c96d206a1 P f2689ba3f5fb4c70bafaa2d05232705a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9dc2ad605434cb49fac1258d87806f5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37367 } } peers { permanent_uuid: "f2689ba3f5fb4c70bafaa2d05232705a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37273 } } peers { permanent_uuid: "09f7fa28cb44464fa809ee1bcf1a2c03" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42613 } }
I20260619 16:46:22.575215 14711 heartbeater.cc:503] Master 127.0.97.62:45429 was elected leader, sending a full tablet report...
I20260619 16:46:22.575780 14732 ts_tablet_manager.cc:1428] T d73d296839b04eedacbdb05c96d206a1 P f2689ba3f5fb4c70bafaa2d05232705a: Time spent starting tablet: real 0.011s user 0.009s sys 0.001s
W20260619 16:46:22.689659 14712 tablet.cc:2369] T d73d296839b04eedacbdb05c96d206a1 P f2689ba3f5fb4c70bafaa2d05232705a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 16:46:22.713294 14579 tablet.cc:2369] T d73d296839b04eedacbdb05c96d206a1 P 09f7fa28cb44464fa809ee1bcf1a2c03: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 16:46:22.742537 14445 tablet.cc:2369] T d73d296839b04eedacbdb05c96d206a1 P a9dc2ad605434cb49fac1258d87806f5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 16:46:22.982662 14737 raft_consensus.cc:482] T d73d296839b04eedacbdb05c96d206a1 P f2689ba3f5fb4c70bafaa2d05232705a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 16:46:22.982937 14737 raft_consensus.cc:504] T d73d296839b04eedacbdb05c96d206a1 P f2689ba3f5fb4c70bafaa2d05232705a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9dc2ad605434cb49fac1258d87806f5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37367 } } peers { permanent_uuid: "f2689ba3f5fb4c70bafaa2d05232705a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37273 } } peers { permanent_uuid: "09f7fa28cb44464fa809ee1bcf1a2c03" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42613 } }
I20260619 16:46:22.984474 14737 leader_election.cc:288] T d73d296839b04eedacbdb05c96d206a1 P f2689ba3f5fb4c70bafaa2d05232705a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a9dc2ad605434cb49fac1258d87806f5 (127.0.97.1:37367), 09f7fa28cb44464fa809ee1bcf1a2c03 (127.0.97.2:42613)
I20260619 16:46:22.990952 14725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:22.984825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:53307 (local address 127.0.97.1:37367)
0619 16:46:22.985028 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:22.985035 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:22.985546 (+ 511us) server_negotiation.cc:407] Connection header received
0619 16:46:22.985715 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:22.985719 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:22.985815 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:22.985890 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:46:22.986479 (+ 589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:22.987188 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:22.989060 (+ 1872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:22.990031 (+ 971us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:22.990171 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:22.990270 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0619 16:46:22.990529 (+ 259us) server_negotiation.cc:299] Negotiation successful
0619 16:46:22.990659 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260619 16:46:22.991146 14738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:22.984768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:37367 (local address 127.0.97.3:53307)
0619 16:46:22.985439 (+ 671us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:22.985454 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 16:46:22.985582 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:46:22.985918 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:46:22.985922 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:46:22.985952 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:46:22.986295 (+ 343us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:22.986307 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:22.987318 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:22.987322 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:22.988906 (+ 1584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:22.988927 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:22.990183 (+ 1256us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:22.990189 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:22.990366 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:22.990401 (+ 35us) client_negotiation.cc:769] Sending connection context
0619 16:46:22.990495 (+ 94us) client_negotiation.cc:240] Negotiation successful
0619 16:46:22.990636 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":562,"mutex_wait_us":63,"thread_start_us":518,"threads_started":1}
I20260619 16:46:22.991930 14728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:22.984921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:33543 (local address 127.0.97.2:42613)
0619 16:46:22.985110 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:22.985116 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:22.985650 (+ 534us) server_negotiation.cc:407] Connection header received
0619 16:46:22.985835 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:22.985839 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:22.985897 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:22.985970 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:46:22.986578 (+ 608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:22.987242 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:22.989679 (+ 2437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:22.990793 (+ 1114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:22.990922 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:22.991014 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0619 16:46:22.991558 (+ 544us) server_negotiation.cc:299] Negotiation successful
0619 16:46:22.991689 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260619 16:46:22.992000 14739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:22.984762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:42613 (local address 127.0.97.3:33543)
0619 16:46:22.985545 (+ 783us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:22.985559 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 16:46:22.985678 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:46:22.985992 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:46:22.985995 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:46:22.986032 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:46:22.986437 (+ 405us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:22.986446 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:22.987364 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:22.987369 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:22.989515 (+ 2146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:22.989529 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:22.991174 (+ 1645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:22.991181 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:22.991363 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:22.991398 (+ 35us) client_negotiation.cc:769] Sending connection context
0619 16:46:22.991518 (+ 120us) client_negotiation.cc:240] Negotiation successful
0619 16:46:22.991646 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":672,"mutex_wait_us":39,"thread_start_us":377,"threads_started":1}
I20260619 16:46:22.992254 14399 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d73d296839b04eedacbdb05c96d206a1" candidate_uuid: "f2689ba3f5fb4c70bafaa2d05232705a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a9dc2ad605434cb49fac1258d87806f5" is_pre_election: true
I20260619 16:46:22.992821 14399 raft_consensus.cc:2454] T d73d296839b04eedacbdb05c96d206a1 P a9dc2ad605434cb49fac1258d87806f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f2689ba3f5fb4c70bafaa2d05232705a in term 0.
I20260619 16:46:22.992914 14533 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d73d296839b04eedacbdb05c96d206a1" candidate_uuid: "f2689ba3f5fb4c70bafaa2d05232705a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09f7fa28cb44464fa809ee1bcf1a2c03" is_pre_election: true
I20260619 16:46:22.993384 14533 raft_consensus.cc:2454] T d73d296839b04eedacbdb05c96d206a1 P 09f7fa28cb44464fa809ee1bcf1a2c03 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f2689ba3f5fb4c70bafaa2d05232705a in term 0.
I20260619 16:46:22.993474 14602 leader_election.cc:302] T d73d296839b04eedacbdb05c96d206a1 P f2689ba3f5fb4c70bafaa2d05232705a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a9dc2ad605434cb49fac1258d87806f5, f2689ba3f5fb4c70bafaa2d05232705a; no voters:
I20260619 16:46:22.993871 14737 raft_consensus.cc:2781] T d73d296839b04eedacbdb05c96d206a1 P f2689ba3f5fb4c70bafaa2d05232705a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 16:46:22.993991 14737 raft_consensus.cc:482] T d73d296839b04eedacbdb05c96d206a1 P f2689ba3f5fb4c70bafaa2d05232705a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 16:46:22.994087 14737 raft_consensus.cc:3037] T d73d296839b04eedacbdb05c96d206a1 P f2689ba3f5fb4c70bafaa2d05232705a [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:46:22.996167 14737 raft_consensus.cc:504] T d73d296839b04eedacbdb05c96d206a1 P f2689ba3f5fb4c70bafaa2d05232705a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9dc2ad605434cb49fac1258d87806f5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37367 } } peers { permanent_uuid: "f2689ba3f5fb4c70bafaa2d05232705a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37273 } } peers { permanent_uuid: "09f7fa28cb44464fa809ee1bcf1a2c03" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42613 } }
I20260619 16:46:22.996778 14737 leader_election.cc:288] T d73d296839b04eedacbdb05c96d206a1 P f2689ba3f5fb4c70bafaa2d05232705a [CANDIDATE]: Term 1 election: Requested vote from peers a9dc2ad605434cb49fac1258d87806f5 (127.0.97.1:37367), 09f7fa28cb44464fa809ee1bcf1a2c03 (127.0.97.2:42613)
I20260619 16:46:22.997268 14399 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d73d296839b04eedacbdb05c96d206a1" candidate_uuid: "f2689ba3f5fb4c70bafaa2d05232705a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a9dc2ad605434cb49fac1258d87806f5"
I20260619 16:46:22.997282 14533 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d73d296839b04eedacbdb05c96d206a1" candidate_uuid: "f2689ba3f5fb4c70bafaa2d05232705a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09f7fa28cb44464fa809ee1bcf1a2c03"
I20260619 16:46:22.997509 14533 raft_consensus.cc:3037] T d73d296839b04eedacbdb05c96d206a1 P 09f7fa28cb44464fa809ee1bcf1a2c03 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:46:22.997501 14399 raft_consensus.cc:3037] T d73d296839b04eedacbdb05c96d206a1 P a9dc2ad605434cb49fac1258d87806f5 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:46:23.000636 14533 raft_consensus.cc:2454] T d73d296839b04eedacbdb05c96d206a1 P 09f7fa28cb44464fa809ee1bcf1a2c03 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f2689ba3f5fb4c70bafaa2d05232705a in term 1.
I20260619 16:46:23.000636 14399 raft_consensus.cc:2454] T d73d296839b04eedacbdb05c96d206a1 P a9dc2ad605434cb49fac1258d87806f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f2689ba3f5fb4c70bafaa2d05232705a in term 1.
I20260619 16:46:23.001053 14602 leader_election.cc:302] T d73d296839b04eedacbdb05c96d206a1 P f2689ba3f5fb4c70bafaa2d05232705a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a9dc2ad605434cb49fac1258d87806f5, f2689ba3f5fb4c70bafaa2d05232705a; no voters:
I20260619 16:46:23.001381 14737 raft_consensus.cc:2781] T d73d296839b04eedacbdb05c96d206a1 P f2689ba3f5fb4c70bafaa2d05232705a [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:46:23.001837 14737 raft_consensus.cc:686] T d73d296839b04eedacbdb05c96d206a1 P f2689ba3f5fb4c70bafaa2d05232705a [term 1 LEADER]: Becoming Leader. State: Replica: f2689ba3f5fb4c70bafaa2d05232705a, State: Running, Role: LEADER
I20260619 16:46:23.002378 14737 consensus_queue.cc:237] T d73d296839b04eedacbdb05c96d206a1 P f2689ba3f5fb4c70bafaa2d05232705a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9dc2ad605434cb49fac1258d87806f5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37367 } } peers { permanent_uuid: "f2689ba3f5fb4c70bafaa2d05232705a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37273 } } peers { permanent_uuid: "09f7fa28cb44464fa809ee1bcf1a2c03" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42613 } }
I20260619 16:46:23.006685 14254 catalog_manager.cc:5495] T d73d296839b04eedacbdb05c96d206a1 P f2689ba3f5fb4c70bafaa2d05232705a reported cstate change: term changed from 0 to 1, leader changed from <none> to f2689ba3f5fb4c70bafaa2d05232705a (127.0.97.3). New cstate: current_term: 1 leader_uuid: "f2689ba3f5fb4c70bafaa2d05232705a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9dc2ad605434cb49fac1258d87806f5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37367 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f2689ba3f5fb4c70bafaa2d05232705a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37273 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "09f7fa28cb44464fa809ee1bcf1a2c03" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42613 } health_report { overall_health: UNKNOWN } } }
I20260619 16:46:23.096437 14747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:23.090856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43538 (local address 127.0.97.3:37273)
0619 16:46:23.091272 (+ 416us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:23.091276 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:23.091424 (+ 148us) server_negotiation.cc:407] Connection header received
0619 16:46:23.091653 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:23.091657 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:23.091760 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:23.091903 (+ 143us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 16:46:23.092311 (+ 408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:23.092983 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:23.094465 (+ 1482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:23.094845 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:23.094945 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:23.095419 (+ 474us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 16:46:23.095909 (+ 490us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 16:46:23.096025 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0619 16:46:23.096117 (+ 92us) server_negotiation.cc:299] Negotiation successful
0619 16:46:23.096185 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":319,"thread_start_us":220,"threads_started":1}
I20260619 16:46:23.104408 14533 raft_consensus.cc:1261] T d73d296839b04eedacbdb05c96d206a1 P 09f7fa28cb44464fa809ee1bcf1a2c03 [term 1 FOLLOWER]: Refusing update from remote peer f2689ba3f5fb4c70bafaa2d05232705a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:46:23.104408 14399 raft_consensus.cc:1261] T d73d296839b04eedacbdb05c96d206a1 P a9dc2ad605434cb49fac1258d87806f5 [term 1 FOLLOWER]: Refusing update from remote peer f2689ba3f5fb4c70bafaa2d05232705a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:46:23.105466 14737 consensus_queue.cc:1035] T d73d296839b04eedacbdb05c96d206a1 P f2689ba3f5fb4c70bafaa2d05232705a [LEADER]: Connected to new peer: Peer: permanent_uuid: "09f7fa28cb44464fa809ee1bcf1a2c03" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42613 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:46:23.105835 14740 consensus_queue.cc:1035] T d73d296839b04eedacbdb05c96d206a1 P f2689ba3f5fb4c70bafaa2d05232705a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a9dc2ad605434cb49fac1258d87806f5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37367 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:46:23.116458 14748 mvcc.cc:205] Tried to move back new op lower bound from 7298611540385402880 to 7298611539981598720. Current Snapshot: MvccSnapshot[applied={T|T < 7298611540385402880}]
I20260619 16:46:23.129690 14254 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42940:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:46:23.129999 14254 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42940:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:46:23.133421 14254 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e1e294d313564ec99776f3b6b38335d5: Sending DeleteTablet for 3 replicas of tablet d73d296839b04eedacbdb05c96d206a1
I20260619 16:46:23.134584 14646 tablet_service.cc:1514] Processing DeleteTablet for tablet d73d296839b04eedacbdb05c96d206a1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:46:23 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:59096
I20260619 16:46:23.134562 14379 tablet_service.cc:1514] Processing DeleteTablet for tablet d73d296839b04eedacbdb05c96d206a1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:46:23 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:47342
I20260619 16:46:23.134748 14513 tablet_service.cc:1514] Processing DeleteTablet for tablet d73d296839b04eedacbdb05c96d206a1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:46:23 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:52230
I20260619 16:46:23.136965 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu pid 14316
I20260619 16:46:23.152040 14761 tablet_replica.cc:330] T d73d296839b04eedacbdb05c96d206a1 P 09f7fa28cb44464fa809ee1bcf1a2c03: stopping tablet replica
I20260619 16:46:23.152555 14761 raft_consensus.cc:2229] T d73d296839b04eedacbdb05c96d206a1 P 09f7fa28cb44464fa809ee1bcf1a2c03 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 16:46:23.153084 14761 raft_consensus.cc:2258] T d73d296839b04eedacbdb05c96d206a1 P 09f7fa28cb44464fa809ee1bcf1a2c03 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:46:23.153390 14760 tablet_replica.cc:330] T d73d296839b04eedacbdb05c96d206a1 P f2689ba3f5fb4c70bafaa2d05232705a: stopping tablet replica
I20260619 16:46:23.153869 14760 raft_consensus.cc:2229] T d73d296839b04eedacbdb05c96d206a1 P f2689ba3f5fb4c70bafaa2d05232705a [term 1 LEADER]: Raft consensus shutting down.
I20260619 16:46:23.154803 14759 tablet_replica.cc:330] T d73d296839b04eedacbdb05c96d206a1 P a9dc2ad605434cb49fac1258d87806f5: stopping tablet replica
I20260619 16:46:23.155295 14759 raft_consensus.cc:2229] T d73d296839b04eedacbdb05c96d206a1 P a9dc2ad605434cb49fac1258d87806f5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 16:46:23.155761 14759 raft_consensus.cc:2258] T d73d296839b04eedacbdb05c96d206a1 P a9dc2ad605434cb49fac1258d87806f5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:46:23.155936 14761 ts_tablet_manager.cc:1905] T d73d296839b04eedacbdb05c96d206a1 P 09f7fa28cb44464fa809ee1bcf1a2c03: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:46:23.156440 14760 raft_consensus.cc:2258] T d73d296839b04eedacbdb05c96d206a1 P f2689ba3f5fb4c70bafaa2d05232705a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:46:23.158113 14759 ts_tablet_manager.cc:1905] T d73d296839b04eedacbdb05c96d206a1 P a9dc2ad605434cb49fac1258d87806f5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:46:23.160287 14760 ts_tablet_manager.cc:1905] T d73d296839b04eedacbdb05c96d206a1 P f2689ba3f5fb4c70bafaa2d05232705a: Deleting tablet data with delete state TABLET_DATA_DELETED
Jun 19 16:46:23 dist-test-slave-wvz7 krb5kdc[14210](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887580, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260619 16:46:23.162889 14761 ts_tablet_manager.cc:1918] T d73d296839b04eedacbdb05c96d206a1 P 09f7fa28cb44464fa809ee1bcf1a2c03: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 16:46:23.163111 14761 log.cc:1196] T d73d296839b04eedacbdb05c96d206a1 P 09f7fa28cb44464fa809ee1bcf1a2c03: Deleting WAL directory at /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887385903558-388-0/minicluster-data/ts-1/wal/wals/d73d296839b04eedacbdb05c96d206a1
I20260619 16:46:23.163590 14761 ts_tablet_manager.cc:1939] T d73d296839b04eedacbdb05c96d206a1 P 09f7fa28cb44464fa809ee1bcf1a2c03: Deleting consensus metadata
I20260619 16:46:23.165494 14240 catalog_manager.cc:4841] TS 09f7fa28cb44464fa809ee1bcf1a2c03 (127.0.97.2:42613): tablet d73d296839b04eedacbdb05c96d206a1 (table test-table [id=f3786b1d03274b5c9ce59a9404170428]) successfully deleted
I20260619 16:46:23.166533 14759 ts_tablet_manager.cc:1918] T d73d296839b04eedacbdb05c96d206a1 P a9dc2ad605434cb49fac1258d87806f5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 16:46:23.166751 14759 log.cc:1196] T d73d296839b04eedacbdb05c96d206a1 P a9dc2ad605434cb49fac1258d87806f5: Deleting WAL directory at /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887385903558-388-0/minicluster-data/ts-0/wal/wals/d73d296839b04eedacbdb05c96d206a1
I20260619 16:46:23.167167 14759 ts_tablet_manager.cc:1939] T d73d296839b04eedacbdb05c96d206a1 P a9dc2ad605434cb49fac1258d87806f5: Deleting consensus metadata
I20260619 16:46:23.170729 14242 catalog_manager.cc:4841] TS a9dc2ad605434cb49fac1258d87806f5 (127.0.97.1:37367): tablet d73d296839b04eedacbdb05c96d206a1 (table test-table [id=f3786b1d03274b5c9ce59a9404170428]) successfully deleted
I20260619 16:46:23.173563 14725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:23.138060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36748 (local address 127.0.97.1:37367)
0619 16:46:23.138245 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:23.138253 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:23.144171 (+ 5918us) server_negotiation.cc:407] Connection header received
0619 16:46:23.145152 (+ 981us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:23.145160 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:23.145292 (+ 132us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:23.145370 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:46:23.149417 (+ 4047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:23.150744 (+ 1327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:23.154343 (+ 3599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:23.155122 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:23.155252 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:23.165902 (+ 10650us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:46:23.165919 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:46:23.165937 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:46:23.165984 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:46:23.169949 (+ 3965us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:23.170961 (+ 1012us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:23.170970 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:23.170973 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:23.171133 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:23.171585 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:23.171590 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:23.171593 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:23.172031 (+ 438us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:46:23.172329 (+ 298us) server_negotiation.cc:1035] Waiting for connection context
0619 16:46:23.172977 (+ 648us) server_negotiation.cc:299] Negotiation successful
0619 16:46:23.173198 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260619 16:46:23.174930 14439 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:46:23.175621 14760 ts_tablet_manager.cc:1918] T d73d296839b04eedacbdb05c96d206a1 P f2689ba3f5fb4c70bafaa2d05232705a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 16:46:23.175824 14760 log.cc:1196] T d73d296839b04eedacbdb05c96d206a1 P f2689ba3f5fb4c70bafaa2d05232705a: Deleting WAL directory at /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887385903558-388-0/minicluster-data/ts-2/wal/wals/d73d296839b04eedacbdb05c96d206a1
I20260619 16:46:23.176393 14760 ts_tablet_manager.cc:1939] T d73d296839b04eedacbdb05c96d206a1 P f2689ba3f5fb4c70bafaa2d05232705a: Deleting consensus metadata
I20260619 16:46:23.178396 14239 catalog_manager.cc:4841] TS f2689ba3f5fb4c70bafaa2d05232705a (127.0.97.3:37273): tablet d73d296839b04eedacbdb05c96d206a1 (table test-table [id=f3786b1d03274b5c9ce59a9404170428]) successfully deleted
I20260619 16:46:23.282888 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu with pid 14316
I20260619 16:46:23.296793 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu pid 14450
Jun 19 16:46:23 dist-test-slave-wvz7 krb5kdc[14210](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887580, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260619 16:46:23.313789 14728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:23.297645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39714 (local address 127.0.97.2:42613)
0619 16:46:23.297926 (+ 281us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:23.297933 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:23.297956 (+ 23us) server_negotiation.cc:407] Connection header received
0619 16:46:23.298096 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:23.298107 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:23.298204 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:23.298379 (+ 175us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:46:23.299563 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:23.300793 (+ 1230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:23.302455 (+ 1662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:23.303090 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:23.303221 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:23.307473 (+ 4252us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:46:23.307493 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:46:23.307513 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:46:23.307557 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:46:23.311114 (+ 3557us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:23.311718 (+ 604us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:23.311727 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:23.311731 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:23.311879 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:23.312182 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:23.312186 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:23.312189 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:23.312657 (+ 468us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:46:23.312960 (+ 303us) server_negotiation.cc:1035] Waiting for connection context
0619 16:46:23.313175 (+ 215us) server_negotiation.cc:299] Negotiation successful
0619 16:46:23.313395 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139}
I20260619 16:46:23.314492 14573 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:46:23.400785 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu with pid 14450
I20260619 16:46:23.420641 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu pid 14583
Jun 19 16:46:23 dist-test-slave-wvz7 krb5kdc[14210](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887580, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260619 16:46:23.438426 14747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:23.421433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43540 (local address 127.0.97.3:37273)
0619 16:46:23.421589 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:23.421593 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:23.421610 (+ 17us) server_negotiation.cc:407] Connection header received
0619 16:46:23.421824 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:23.421828 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:23.421891 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:23.421983 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:46:23.423610 (+ 1627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:23.424391 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:23.426269 (+ 1878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:23.426705 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:23.426807 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:23.431934 (+ 5127us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:46:23.431948 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:46:23.431966 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:46:23.432002 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:46:23.434872 (+ 2870us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:23.435725 (+ 853us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:23.435734 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:23.435737 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:23.435911 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:23.436350 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:23.436355 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:23.436358 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:23.436907 (+ 549us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:46:23.437225 (+ 318us) server_negotiation.cc:1035] Waiting for connection context
0619 16:46:23.437542 (+ 317us) server_negotiation.cc:299] Negotiation successful
0619 16:46:23.437879 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260619 16:46:23.439317 14706 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:46:23.559352 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu with pid 14583
I20260619 16:46:23.574180 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu pid 14223
I20260619 16:46:23.575345 14285 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:46:23.682782 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu with pid 14223
I20260619 16:46:23.699621 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu
/tmp/dist-test-taskEs2EH8/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887385903558-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887385903558-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887385903558-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887385903558-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45429
--webserver_interface=127.0.97.62
--webserver_port=34959
--keytab_file=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887385903558-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38239
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:45429
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887385903558-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887385903558-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887385903558-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:46:24.016460 14767 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:46:24.016742 14767 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:46:24.016829 14767 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:46:24.023983 14767 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:46:24.024096 14767 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:46:24.024140 14767 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:46:24.024194 14767 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:46:24.024235 14767 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:46:24.033968 14767 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38239
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887385903558-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887385903558-388-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:45429
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887385903558-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45429
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887385903558-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=34959
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887385903558-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:46:24.035204 14767 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:46:24.036417 14767 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:46:24.057142 14773 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:46:24.057168 14774 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:46:24.057194 14776 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:46:24.058007 14767 server_base.cc:1034] running on GCE node
I20260619 16:46:24.058861 14767 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:46:24.060350 14767 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:46:24.061578 14767 hybrid_clock.cc:648] HybridClock initialized: now 1781887584061522 us; error 44 us; skew 500 ppm
I20260619 16:46:24.061982 14767 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:46:24 dist-test-slave-wvz7 krb5kdc[14210](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887584, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:46:24.065471 14767 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260619 16:46:24.067001 14767 webserver.cc:460] Webserver started at http://127.0.97.62:34959/ using document root <none> and password file <none>
I20260619 16:46:24.067682 14767 fs_manager.cc:359] Metadata directory not provided
I20260619 16:46:24.067790 14767 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:46:24.073401 14767 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.000s
I20260619 16:46:24.076202 14783 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:46:24.077526 14767 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260619 16:46:24.077674 14767 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887385903558-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887385903558-388-0/minicluster-data/master-0/wal
uuid: "e1e294d313564ec99776f3b6b38335d5"
format_stamp: "Formatted at 2026-06-19 16:46:20 on dist-test-slave-wvz7"
server_key: "fc92fef4748c44f2be23bd7ef93fe4f1"
server_key_iv: "529772319276df9c93bee0a3ed2a668d"
server_key_version: "encryptionkey@0"
I20260619 16:46:24.078691 14767 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887385903558-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887385903558-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887385903558-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:46:24.095569 14767 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:46:24.099749 14767 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:46:24.100018 14767 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:46:24.117048 14767 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:45429
I20260619 16:46:24.117069 14834 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:45429 every 8 connection(s)
I20260619 16:46:24.118551 14767 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887385903558-388-0/minicluster-data/master-0/data/info.pb
I20260619 16:46:24.119992 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu as pid 14767
I20260619 16:46:24.120728 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu
/tmp/dist-test-taskEs2EH8/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887385903558-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887385903558-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887385903558-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887385903558-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:37367
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:45429
--webserver_port=42735
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887385903558-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38239
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887385903558-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887385903558-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887385903558-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 16:46:24.125528 14835 sys_catalog.cc:263] Verifying existing consensus state
I20260619 16:46:24.129588 14835 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e1e294d313564ec99776f3b6b38335d5: Bootstrap starting.
I20260619 16:46:24.157683 14835 log.cc:824] T 00000000000000000000000000000000 P e1e294d313564ec99776f3b6b38335d5: Log is configured to *not* fsync() on all Append() calls
I20260619 16:46:24.173102 14835 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e1e294d313564ec99776f3b6b38335d5: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 16:46:24.173803 14835 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e1e294d313564ec99776f3b6b38335d5: Bootstrap complete.
I20260619 16:46:24.187868 14835 raft_consensus.cc:348] T 00000000000000000000000000000000 P e1e294d313564ec99776f3b6b38335d5 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1e294d313564ec99776f3b6b38335d5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45429 } }
I20260619 16:46:24.189179 14835 raft_consensus.cc:729] T 00000000000000000000000000000000 P e1e294d313564ec99776f3b6b38335d5 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1e294d313564ec99776f3b6b38335d5, State: Initialized, Role: FOLLOWER
I20260619 16:46:24.190006 14835 consensus_queue.cc:260] T 00000000000000000000000000000000 P e1e294d313564ec99776f3b6b38335d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1e294d313564ec99776f3b6b38335d5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45429 } }
I20260619 16:46:24.190302 14835 raft_consensus.cc:388] T 00000000000000000000000000000000 P e1e294d313564ec99776f3b6b38335d5 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:46:24.190438 14835 raft_consensus.cc:482] T 00000000000000000000000000000000 P e1e294d313564ec99776f3b6b38335d5 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:46:24.190608 14835 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e1e294d313564ec99776f3b6b38335d5 [term 1 FOLLOWER]: Advancing to term 2
I20260619 16:46:24.196053 14835 raft_consensus.cc:504] T 00000000000000000000000000000000 P e1e294d313564ec99776f3b6b38335d5 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1e294d313564ec99776f3b6b38335d5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45429 } }
I20260619 16:46:24.196652 14835 leader_election.cc:302] T 00000000000000000000000000000000 P e1e294d313564ec99776f3b6b38335d5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e1e294d313564ec99776f3b6b38335d5; no voters:
I20260619 16:46:24.197309 14835 leader_election.cc:288] T 00000000000000000000000000000000 P e1e294d313564ec99776f3b6b38335d5 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260619 16:46:24.197476 14839 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e1e294d313564ec99776f3b6b38335d5 [term 2 FOLLOWER]: Leader election won for term 2
I20260619 16:46:24.198549 14839 raft_consensus.cc:686] T 00000000000000000000000000000000 P e1e294d313564ec99776f3b6b38335d5 [term 2 LEADER]: Becoming Leader. State: Replica: e1e294d313564ec99776f3b6b38335d5, State: Running, Role: LEADER
I20260619 16:46:24.199034 14839 consensus_queue.cc:237] T 00000000000000000000000000000000 P e1e294d313564ec99776f3b6b38335d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1e294d313564ec99776f3b6b38335d5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45429 } }
I20260619 16:46:24.199399 14835 sys_catalog.cc:564] T 00000000000000000000000000000000 P e1e294d313564ec99776f3b6b38335d5 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:46:24.201283 14841 sys_catalog.cc:455] T 00000000000000000000000000000000 P e1e294d313564ec99776f3b6b38335d5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e1e294d313564ec99776f3b6b38335d5. Latest consensus state: current_term: 2 leader_uuid: "e1e294d313564ec99776f3b6b38335d5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1e294d313564ec99776f3b6b38335d5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45429 } } }
I20260619 16:46:24.201627 14841 sys_catalog.cc:458] T 00000000000000000000000000000000 P e1e294d313564ec99776f3b6b38335d5 [sys.catalog]: This master's current role is: LEADER
I20260619 16:46:24.202104 14840 sys_catalog.cc:455] T 00000000000000000000000000000000 P e1e294d313564ec99776f3b6b38335d5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "e1e294d313564ec99776f3b6b38335d5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1e294d313564ec99776f3b6b38335d5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45429 } } }
I20260619 16:46:24.202363 14840 sys_catalog.cc:458] T 00000000000000000000000000000000 P e1e294d313564ec99776f3b6b38335d5 [sys.catalog]: This master's current role is: LEADER
I20260619 16:46:24.212452 14845 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:46:24.219790 14845 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:46:24.221171 14845 catalog_manager.cc:1259] Loaded cluster ID: b3db8dcb2ff3459f9a5b16ab9e88649d
I20260619 16:46:24.221268 14845 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:46:24.224202 14845 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260619 16:46:24.241750 14845 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260619 16:46:24.264238 14845 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:46:24.266469 14845 catalog_manager.cc:5879] T 00000000000000000000000000000000 P e1e294d313564ec99776f3b6b38335d5: Loaded TSK: 0
I20260619 16:46:24.268016 14845 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260619 16:46:24.480202 14837 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:46:24.480449 14837 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:46:24.480576 14837 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:46:24.487689 14837 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:46:24.487789 14837 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:46:24.487926 14837 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260619 16:46:24.497509 14837 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38239
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887385903558-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887385903558-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887385903558-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:37367
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887385903558-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=42735
--tserver_master_addrs=127.0.97.62:45429
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887385903558-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:46:24.498787 14837 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:46:24.500058 14837 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:46:24.514170 14862 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:46:24.514178 14863 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:46:24.514695 14865 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:46:24.514967 14837 server_base.cc:1034] running on GCE node
I20260619 16:46:24.515504 14837 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:46:24.516403 14837 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:46:24.517593 14837 hybrid_clock.cc:648] HybridClock initialized: now 1781887584517534 us; error 54 us; skew 500 ppm
I20260619 16:46:24.517920 14837 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:46:24 dist-test-slave-wvz7 krb5kdc[14210](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887584, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:46:24.520823 14837 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260619 16:46:24.522221 14837 webserver.cc:460] Webserver started at http://127.0.97.1:42735/ using document root <none> and password file <none>
I20260619 16:46:24.522848 14837 fs_manager.cc:359] Metadata directory not provided
I20260619 16:46:24.522970 14837 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:46:24.528286 14837 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 16:46:24.530987 14872 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:46:24.532366 14837 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 16:46:24.532508 14837 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887385903558-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887385903558-388-0/minicluster-data/ts-0/wal
uuid: "a9dc2ad605434cb49fac1258d87806f5"
format_stamp: "Formatted at 2026-06-19 16:46:21 on dist-test-slave-wvz7"
server_key: "a1fdf62f258763c823e087b9c5d87f23"
server_key_iv: "8a28126bc939a78919f716c3739ca8da"
server_key_version: "encryptionkey@0"
I20260619 16:46:24.533435 14837 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887385903558-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887385903558-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887385903558-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:46:24.546928 14837 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:46:24.551060 14837 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:46:24.551306 14837 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:46:24.552276 14837 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:46:24.553562 14837 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:46:24.553658 14837 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:46:24.553776 14837 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:46:24.553846 14837 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:46:24.584226 14837 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:37367
I20260619 16:46:24.584278 14984 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:37367 every 8 connection(s)
I20260619 16:46:24.585572 14837 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887385903558-388-0/minicluster-data/ts-0/data/info.pb
I20260619 16:46:24.595182 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu as pid 14837
I20260619 16:46:24.595719 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu
/tmp/dist-test-taskEs2EH8/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887385903558-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887385903558-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887385903558-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887385903558-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:42613
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:45429
--webserver_port=35761
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887385903558-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38239
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887385903558-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887385903558-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887385903558-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 16:46:24 dist-test-slave-wvz7 krb5kdc[14210](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887584, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 16:46:24.607606 14988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:24.588471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45429 (local address 127.0.97.1:51421)
0619 16:46:24.589268 (+ 797us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:24.589312 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0619 16:46:24.590989 (+ 1677us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:46:24.592259 (+ 1270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:46:24.592267 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:46:24.592742 (+ 475us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:46:24.593259 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:24.593276 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:24.594877 (+ 1601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:24.594881 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:24.596278 (+ 1397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:24.596292 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:24.597279 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:24.597286 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:24.597537 (+ 251us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:24.598144 (+ 607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:46:24.598167 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:46:24.600402 (+ 2235us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:46:24.603571 (+ 3169us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:46:24.603580 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:46:24.603595 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:46:24.603991 (+ 396us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:46:24.604392 (+ 401us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:46:24.604397 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:46:24.604401 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:46:24.604570 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:46:24.605240 (+ 670us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:46:24.605247 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:46:24.605556 (+ 309us) client_negotiation.cc:769] Sending connection context
0619 16:46:24.605831 (+ 275us) client_negotiation.cc:240] Negotiation successful
0619 16:46:24.606348 (+ 517us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":524,"thread_start_us":322,"threads_started":1}
I20260619 16:46:24.607606 14987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:24.588215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:51421 (local address 127.0.97.62:45429)
0619 16:46:24.588744 (+ 529us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:24.588756 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:24.589432 (+ 676us) server_negotiation.cc:407] Connection header received
0619 16:46:24.591508 (+ 2076us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:24.591521 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:24.591855 (+ 334us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:24.592137 (+ 282us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:46:24.593449 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:24.594713 (+ 1264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:24.596418 (+ 1705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:24.597064 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:24.597245 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:24.600654 (+ 3409us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:46:24.600661 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:46:24.600697 (+ 36us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:46:24.600743 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:46:24.603445 (+ 2702us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:24.604111 (+ 666us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:24.604117 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:24.604122 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:24.604278 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:24.604673 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:24.604677 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:24.604679 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:24.605113 (+ 434us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:46:24.605283 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0619 16:46:24.606021 (+ 738us) server_negotiation.cc:299] Negotiation successful
0619 16:46:24.606381 (+ 360us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":347,"thread_start_us":219,"threads_started":1}
I20260619 16:46:24.610680 14985 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45429
I20260619 16:46:24.610948 14985 heartbeater.cc:464] Registering TS with master...
I20260619 16:46:24.611647 14985 heartbeater.cc:511] Master 127.0.97.62:45429 requested a full tablet report, sending...
I20260619 16:46:24.613909 14800 ts_manager.cc:194] Registered new tserver with Master: a9dc2ad605434cb49fac1258d87806f5 (127.0.97.1:37367)
I20260619 16:46:24.615999 14800 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:51421
W20260619 16:46:24.914919 14989 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:46:24.915216 14989 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:46:24.915313 14989 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:46:24.922945 14989 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:46:24.923089 14989 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:46:24.923264 14989 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260619 16:46:24.933926 14989 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38239
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887385903558-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887385903558-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887385903558-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:42613
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887385903558-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=35761
--tserver_master_addrs=127.0.97.62:45429
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887385903558-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:46:24.935191 14989 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:46:24.936460 14989 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:46:24.949364 14994 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:46:24.949306 14995 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:46:24.949342 14997 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:46:24.950048 14989 server_base.cc:1034] running on GCE node
I20260619 16:46:24.950608 14989 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:46:24.951573 14989 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:46:24.952800 14989 hybrid_clock.cc:648] HybridClock initialized: now 1781887584952734 us; error 53 us; skew 500 ppm
I20260619 16:46:24.953123 14989 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:46:24 dist-test-slave-wvz7 krb5kdc[14210](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887584, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:46:24.956277 14989 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260619 16:46:24.957656 14989 webserver.cc:460] Webserver started at http://127.0.97.2:35761/ using document root <none> and password file <none>
I20260619 16:46:24.958333 14989 fs_manager.cc:359] Metadata directory not provided
I20260619 16:46:24.958462 14989 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:46:24.963474 14989 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.003s
I20260619 16:46:24.965994 15004 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:46:24.967300 14989 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260619 16:46:24.967455 14989 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887385903558-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887385903558-388-0/minicluster-data/ts-1/wal
uuid: "09f7fa28cb44464fa809ee1bcf1a2c03"
format_stamp: "Formatted at 2026-06-19 16:46:21 on dist-test-slave-wvz7"
server_key: "f18fa1c04ddfbf586c6a8778a22869b0"
server_key_iv: "7dfdcd84e637eff4f21842baa7227bf2"
server_key_version: "encryptionkey@0"
I20260619 16:46:24.968369 14989 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887385903558-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887385903558-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887385903558-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:46:24.990195 14989 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:46:24.993950 14989 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:46:24.994184 14989 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:46:24.995095 14989 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:46:24.996408 14989 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:46:24.996505 14989 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:46:24.996609 14989 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:46:24.996676 14989 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:46:25.025923 14989 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:42613
I20260619 16:46:25.025969 15116 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:42613 every 8 connection(s)
I20260619 16:46:25.027372 14989 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887385903558-388-0/minicluster-data/ts-1/data/info.pb
I20260619 16:46:25.036952 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu as pid 14989
I20260619 16:46:25.037657 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu
/tmp/dist-test-taskEs2EH8/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887385903558-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887385903558-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887385903558-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887385903558-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:37273
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:45429
--webserver_port=42761
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887385903558-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38239
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887385903558-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887385903558-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887385903558-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 16:46:25 dist-test-slave-wvz7 krb5kdc[14210](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887584, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 16:46:25.046741 14987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:25.029823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:60937 (local address 127.0.97.62:45429)
0619 16:46:25.030016 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:25.030022 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:25.031194 (+ 1172us) server_negotiation.cc:407] Connection header received
0619 16:46:25.032927 (+ 1733us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:25.032932 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:25.033000 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:25.033089 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:46:25.034535 (+ 1446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:25.035222 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:25.036686 (+ 1464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:25.037177 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:25.037295 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:25.041208 (+ 3913us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:46:25.041214 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:46:25.041217 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:46:25.041251 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:46:25.043724 (+ 2473us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:25.044453 (+ 729us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:25.044459 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:25.044462 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:25.044616 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:25.045009 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:25.045014 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:25.045016 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:25.045365 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:46:25.045500 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0619 16:46:25.046188 (+ 688us) server_negotiation.cc:299] Negotiation successful
0619 16:46:25.046433 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260619 16:46:25.047511 15119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:25.030358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45429 (local address 127.0.97.2:60937)
0619 16:46:25.031034 (+ 676us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:25.031078 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0619 16:46:25.032664 (+ 1586us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:46:25.033269 (+ 605us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:46:25.033277 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:46:25.033796 (+ 519us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:46:25.034387 (+ 591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:25.034407 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:25.035334 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:25.035338 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:25.036555 (+ 1217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:25.036571 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:25.037395 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:25.037404 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:25.037676 (+ 272us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:25.038373 (+ 697us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:46:25.038406 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:46:25.040986 (+ 2580us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:46:25.043866 (+ 2880us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:46:25.043875 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:46:25.043899 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:46:25.044321 (+ 422us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:46:25.044715 (+ 394us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:46:25.044718 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:46:25.044720 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:46:25.044905 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:46:25.045475 (+ 570us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:46:25.045481 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:46:25.045826 (+ 345us) client_negotiation.cc:769] Sending connection context
0619 16:46:25.046063 (+ 237us) client_negotiation.cc:240] Negotiation successful
0619 16:46:25.046443 (+ 380us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":436,"thread_start_us":265,"threads_started":1}
I20260619 16:46:25.048825 15117 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45429
I20260619 16:46:25.049146 15117 heartbeater.cc:464] Registering TS with master...
I20260619 16:46:25.050024 15117 heartbeater.cc:511] Master 127.0.97.62:45429 requested a full tablet report, sending...
I20260619 16:46:25.051542 14800 ts_manager.cc:194] Registered new tserver with Master: 09f7fa28cb44464fa809ee1bcf1a2c03 (127.0.97.2:42613)
I20260619 16:46:25.052320 14800 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:60937
W20260619 16:46:25.348706 15120 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:46:25.348970 15120 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:46:25.349086 15120 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:46:25.356305 15120 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:46:25.356412 15120 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:46:25.356557 15120 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260619 16:46:25.366182 15120 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38239
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887385903558-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887385903558-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887385903558-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:37273
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887385903558-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=42761
--tserver_master_addrs=127.0.97.62:45429
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887385903558-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:46:25.367419 15120 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:46:25.368640 15120 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:46:25.380975 15125 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:46:25.380916 15128 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:46:25.380916 15126 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:46:25.381812 15120 server_base.cc:1034] running on GCE node
I20260619 16:46:25.382336 15120 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:46:25.383438 15120 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:46:25.384642 15120 hybrid_clock.cc:648] HybridClock initialized: now 1781887585384591 us; error 51 us; skew 500 ppm
I20260619 16:46:25.384932 15120 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:46:25 dist-test-slave-wvz7 krb5kdc[14210](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887585, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:46:25.387959 15120 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260619 16:46:25.389348 15120 webserver.cc:460] Webserver started at http://127.0.97.3:42761/ using document root <none> and password file <none>
I20260619 16:46:25.390039 15120 fs_manager.cc:359] Metadata directory not provided
I20260619 16:46:25.390156 15120 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:46:25.395211 15120 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 16:46:25.397893 15135 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:46:25.399173 15120 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 16:46:25.399310 15120 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887385903558-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887385903558-388-0/minicluster-data/ts-2/wal
uuid: "f2689ba3f5fb4c70bafaa2d05232705a"
format_stamp: "Formatted at 2026-06-19 16:46:22 on dist-test-slave-wvz7"
server_key: "f8f2beb316fac04fe57535d8da6ec7d5"
server_key_iv: "1142e95cb44a1e97d33f4af87f3b0bf4"
server_key_version: "encryptionkey@0"
I20260619 16:46:25.400259 15120 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887385903558-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887385903558-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887385903558-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:46:25.456005 15120 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:46:25.459934 15120 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:46:25.460160 15120 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:46:25.461135 15120 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:46:25.462459 15120 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:46:25.462549 15120 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:46:25.462644 15120 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:46:25.462703 15120 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:46:25.492471 15120 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:37273
I20260619 16:46:25.492542 15247 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:37273 every 8 connection(s)
I20260619 16:46:25.493863 15120 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887385903558-388-0/minicluster-data/ts-2/data/info.pb
I20260619 16:46:25.501157 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu as pid 15120
Jun 19 16:46:25 dist-test-slave-wvz7 krb5kdc[14210](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887585, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 16:46:25.514719 14987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:25.496347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:51211 (local address 127.0.97.62:45429)
0619 16:46:25.496587 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:25.496594 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:25.497803 (+ 1209us) server_negotiation.cc:407] Connection header received
0619 16:46:25.499538 (+ 1735us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:25.499543 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:25.499611 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:25.499697 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:46:25.501274 (+ 1577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:25.502213 (+ 939us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:25.504215 (+ 2002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:25.504838 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:25.504935 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:25.508403 (+ 3468us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:46:25.508409 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:46:25.508414 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:46:25.508453 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:46:25.511431 (+ 2978us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:25.512264 (+ 833us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:25.512268 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:25.512270 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:25.512399 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:25.512880 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:25.512883 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:25.512885 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:25.513157 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:46:25.513268 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0619 16:46:25.514171 (+ 903us) server_negotiation.cc:299] Negotiation successful
0619 16:46:25.514381 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260619 16:46:25.515884 15250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:25.496855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45429 (local address 127.0.97.3:51211)
0619 16:46:25.497619 (+ 764us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:25.497664 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0619 16:46:25.499286 (+ 1622us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:46:25.499970 (+ 684us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:46:25.499978 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:46:25.500482 (+ 504us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:46:25.501095 (+ 613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:25.501113 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:25.502459 (+ 1346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:25.502465 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:25.504019 (+ 1554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:25.504032 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:25.505031 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:25.505038 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:25.505319 (+ 281us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:25.506016 (+ 697us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:46:25.506046 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:46:25.508184 (+ 2138us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:46:25.511557 (+ 3373us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:46:25.511564 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:46:25.511576 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:46:25.512151 (+ 575us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:46:25.512509 (+ 358us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:46:25.512512 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:46:25.512515 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:46:25.512760 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:46:25.513280 (+ 520us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:46:25.513288 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:46:25.513721 (+ 433us) client_negotiation.cc:769] Sending connection context
0619 16:46:25.514050 (+ 329us) client_negotiation.cc:240] Negotiation successful
0619 16:46:25.514462 (+ 412us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":518,"thread_start_us":299,"threads_started":1}
I20260619 16:46:25.516949 15251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:25.502095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55766 (local address 127.0.97.62:45429)
0619 16:46:25.502594 (+ 499us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:25.502599 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:25.502871 (+ 272us) server_negotiation.cc:407] Connection header received
0619 16:46:25.503000 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:25.503005 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:25.503076 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:25.503180 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:46:25.504854 (+ 1674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:25.505886 (+ 1032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:25.507180 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:25.507647 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:25.507764 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:25.510887 (+ 3123us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:46:25.510892 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:46:25.510895 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:46:25.510935 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:46:25.513937 (+ 3002us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:25.514761 (+ 824us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:25.514765 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:25.514768 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:25.514910 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:25.515265 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:25.515270 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:25.515273 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:25.515590 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:46:25.515867 (+ 277us) server_negotiation.cc:1035] Waiting for connection context
0619 16:46:25.516290 (+ 423us) server_negotiation.cc:299] Negotiation successful
0619 16:46:25.516653 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":376,"thread_start_us":260,"threads_started":1}
I20260619 16:46:25.517347 15248 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45429
I20260619 16:46:25.517635 15248 heartbeater.cc:464] Registering TS with master...
I20260619 16:46:25.518337 15248 heartbeater.cc:511] Master 127.0.97.62:45429 requested a full tablet report, sending...
I20260619 16:46:25.519588 14800 ts_manager.cc:194] Registered new tserver with Master: f2689ba3f5fb4c70bafaa2d05232705a (127.0.97.3:37273)
I20260619 16:46:25.520360 14800 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:51211
I20260619 16:46:25.531733 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 16:46:25.538612 15251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:25.532649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55774 (local address 127.0.97.62:45429)
0619 16:46:25.532895 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:25.532901 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:25.533321 (+ 420us) server_negotiation.cc:407] Connection header received
0619 16:46:25.533540 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:25.533543 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:25.533680 (+ 137us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:25.533844 (+ 164us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 16:46:25.534239 (+ 395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:25.534952 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:25.536348 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:25.536841 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:25.536971 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:25.537483 (+ 512us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 16:46:25.538104 (+ 621us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 16:46:25.538221 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0619 16:46:25.538342 (+ 121us) server_negotiation.cc:299] Negotiation successful
0619 16:46:25.538412 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97}
I20260619 16:46:25.539539 14800 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:55774:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 16:46:25.546579 14800 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 16:46:25.592410 15260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:25.558441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:37273 (local address 127.0.0.1:43556)
0619 16:46:25.582895 (+ 24454us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:25.582911 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 16:46:25.583013 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:46:25.584602 (+ 1589us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:46:25.584609 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:46:25.584637 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:46:25.585177 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:25.585196 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:25.587540 (+ 2344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:25.587545 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:25.589930 (+ 2385us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:25.589945 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:25.591710 (+ 1765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:25.591714 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:25.591835 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:25.591870 (+ 35us) client_negotiation.cc:769] Sending connection context
0619 16:46:25.591949 (+ 79us) client_negotiation.cc:240] Negotiation successful
0619 16:46:25.592078 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24335,"thread_start_us":24206,"threads_started":1}
I20260619 16:46:25.593109 15263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:25.559052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43556 (local address 127.0.97.3:37273)
0619 16:46:25.583756 (+ 24704us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:25.583764 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:25.583910 (+ 146us) server_negotiation.cc:407] Connection header received
0619 16:46:25.584021 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:25.584029 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:25.584307 (+ 278us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:25.585410 (+ 1103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:46:25.585682 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:25.587199 (+ 1517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:25.590125 (+ 2926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:25.591514 (+ 1389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:25.592298 (+ 784us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:25.592541 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0619 16:46:25.592606 (+ 65us) server_negotiation.cc:299] Negotiation successful
0619 16:46:25.592760 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24542,"thread_start_us":28241,"threads_started":1}
I20260619 16:46:25.594285 15261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:25.558260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:37367 (local address 127.0.0.1:36750)
0619 16:46:25.581923 (+ 23663us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:25.581973 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0619 16:46:25.582189 (+ 216us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:46:25.585550 (+ 3361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:46:25.585556 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:46:25.585586 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:46:25.585914 (+ 328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:25.585925 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:25.588801 (+ 2876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:25.588806 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:25.591480 (+ 2674us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:25.591496 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:25.593457 (+ 1961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:25.593463 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:25.593620 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:25.593655 (+ 35us) client_negotiation.cc:769] Sending connection context
0619 16:46:25.593719 (+ 64us) client_negotiation.cc:240] Negotiation successful
0619 16:46:25.593879 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23332,"mutex_wait_us":62,"thread_start_us":26769,"threads_started":1}
I20260619 16:46:25.594671 15259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:25.558680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:42613 (local address 127.0.0.1:39716)
0619 16:46:25.585320 (+ 26640us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:25.585336 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 16:46:25.585511 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:46:25.586389 (+ 878us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:46:25.586393 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:46:25.586416 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:46:25.587408 (+ 992us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:25.587421 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:25.589273 (+ 1852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:25.589277 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:25.592087 (+ 2810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:25.592101 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:25.593678 (+ 1577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:25.593685 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:25.593877 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:25.593899 (+ 22us) client_negotiation.cc:769] Sending connection context
0619 16:46:25.593973 (+ 74us) client_negotiation.cc:240] Negotiation successful
0619 16:46:25.594092 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26535,"thread_start_us":24396,"threads_started":1}
I20260619 16:46:25.595225 15262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:25.559136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39716 (local address 127.0.97.2:42613)
0619 16:46:25.581069 (+ 21933us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:25.581080 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:25.585416 (+ 4336us) server_negotiation.cc:407] Connection header received
0619 16:46:25.585689 (+ 273us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:25.585696 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:25.585963 (+ 267us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:25.586125 (+ 162us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:46:25.587581 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:25.589095 (+ 1514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:25.592251 (+ 3156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:25.593477 (+ 1226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:25.593588 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:25.593789 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0619 16:46:25.594035 (+ 246us) server_negotiation.cc:299] Negotiation successful
0619 16:46:25.594184 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21734,"thread_start_us":21952,"threads_started":1}
I20260619 16:46:25.595340 15258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:25.558275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36750 (local address 127.0.97.1:37367)
0619 16:46:25.584377 (+ 26102us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:25.584385 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:25.584410 (+ 25us) server_negotiation.cc:407] Connection header received
0619 16:46:25.584497 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:25.584504 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:25.584734 (+ 230us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:25.586107 (+ 1373us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:46:25.586797 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:25.588644 (+ 1847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:25.591659 (+ 3015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:25.593233 (+ 1574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:25.594608 (+ 1375us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:25.594833 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0619 16:46:25.594895 (+ 62us) server_negotiation.cc:299] Negotiation successful
0619 16:46:25.595049 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25951,"thread_start_us":26980,"threads_started":1}
I20260619 16:46:25.597276 15183 tablet_service.cc:1467] Processing CreateTablet for tablet 700c272e6d53403ab4bbee7c4b1a630c (DEFAULT_TABLE table=test-table [id=bbf7ff42f11c4f439078d9a9d22f6d18]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:46:25.598480 15183 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 700c272e6d53403ab4bbee7c4b1a630c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:46:25.598527 15052 tablet_service.cc:1467] Processing CreateTablet for tablet 700c272e6d53403ab4bbee7c4b1a630c (DEFAULT_TABLE table=test-table [id=bbf7ff42f11c4f439078d9a9d22f6d18]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:46:25.598524 14920 tablet_service.cc:1467] Processing CreateTablet for tablet 700c272e6d53403ab4bbee7c4b1a630c (DEFAULT_TABLE table=test-table [id=bbf7ff42f11c4f439078d9a9d22f6d18]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:46:25.599766 14920 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 700c272e6d53403ab4bbee7c4b1a630c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:46:25.599773 15052 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 700c272e6d53403ab4bbee7c4b1a630c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:46:25.610392 15264 tablet_bootstrap.cc:492] T 700c272e6d53403ab4bbee7c4b1a630c P f2689ba3f5fb4c70bafaa2d05232705a: Bootstrap starting.
I20260619 16:46:25.613682 15264 tablet_bootstrap.cc:654] T 700c272e6d53403ab4bbee7c4b1a630c P f2689ba3f5fb4c70bafaa2d05232705a: Neither blocks nor log segments found. Creating new log.
I20260619 16:46:25.614524 15266 tablet_bootstrap.cc:492] T 700c272e6d53403ab4bbee7c4b1a630c P 09f7fa28cb44464fa809ee1bcf1a2c03: Bootstrap starting.
I20260619 16:46:25.614977 15264 log.cc:824] T 700c272e6d53403ab4bbee7c4b1a630c P f2689ba3f5fb4c70bafaa2d05232705a: Log is configured to *not* fsync() on all Append() calls
I20260619 16:46:25.616334 15265 tablet_bootstrap.cc:492] T 700c272e6d53403ab4bbee7c4b1a630c P a9dc2ad605434cb49fac1258d87806f5: Bootstrap starting.
I20260619 16:46:25.617925 15264 tablet_bootstrap.cc:492] T 700c272e6d53403ab4bbee7c4b1a630c P f2689ba3f5fb4c70bafaa2d05232705a: No bootstrap required, opened a new log
I20260619 16:46:25.618175 15264 ts_tablet_manager.cc:1397] T 700c272e6d53403ab4bbee7c4b1a630c P f2689ba3f5fb4c70bafaa2d05232705a: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.006s
I20260619 16:46:25.618340 15266 tablet_bootstrap.cc:654] T 700c272e6d53403ab4bbee7c4b1a630c P 09f7fa28cb44464fa809ee1bcf1a2c03: Neither blocks nor log segments found. Creating new log.
I20260619 16:46:25.618960 14985 heartbeater.cc:503] Master 127.0.97.62:45429 was elected leader, sending a full tablet report...
I20260619 16:46:25.619637 15266 log.cc:824] T 700c272e6d53403ab4bbee7c4b1a630c P 09f7fa28cb44464fa809ee1bcf1a2c03: Log is configured to *not* fsync() on all Append() calls
I20260619 16:46:25.619706 15265 tablet_bootstrap.cc:654] T 700c272e6d53403ab4bbee7c4b1a630c P a9dc2ad605434cb49fac1258d87806f5: Neither blocks nor log segments found. Creating new log.
I20260619 16:46:25.620939 15265 log.cc:824] T 700c272e6d53403ab4bbee7c4b1a630c P a9dc2ad605434cb49fac1258d87806f5: Log is configured to *not* fsync() on all Append() calls
I20260619 16:46:25.626331 15266 tablet_bootstrap.cc:492] T 700c272e6d53403ab4bbee7c4b1a630c P 09f7fa28cb44464fa809ee1bcf1a2c03: No bootstrap required, opened a new log
I20260619 16:46:25.626579 15266 ts_tablet_manager.cc:1397] T 700c272e6d53403ab4bbee7c4b1a630c P 09f7fa28cb44464fa809ee1bcf1a2c03: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.005s
I20260619 16:46:25.626665 15265 tablet_bootstrap.cc:492] T 700c272e6d53403ab4bbee7c4b1a630c P a9dc2ad605434cb49fac1258d87806f5: No bootstrap required, opened a new log
I20260619 16:46:25.626921 15265 ts_tablet_manager.cc:1397] T 700c272e6d53403ab4bbee7c4b1a630c P a9dc2ad605434cb49fac1258d87806f5: Time spent bootstrapping tablet: real 0.011s user 0.001s sys 0.008s
I20260619 16:46:25.627732 15264 raft_consensus.cc:348] T 700c272e6d53403ab4bbee7c4b1a630c P f2689ba3f5fb4c70bafaa2d05232705a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9dc2ad605434cb49fac1258d87806f5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37367 } } peers { permanent_uuid: "f2689ba3f5fb4c70bafaa2d05232705a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37273 } } peers { permanent_uuid: "09f7fa28cb44464fa809ee1bcf1a2c03" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42613 } }
I20260619 16:46:25.628158 15264 raft_consensus.cc:374] T 700c272e6d53403ab4bbee7c4b1a630c P f2689ba3f5fb4c70bafaa2d05232705a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:46:25.628260 15264 raft_consensus.cc:729] T 700c272e6d53403ab4bbee7c4b1a630c P f2689ba3f5fb4c70bafaa2d05232705a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f2689ba3f5fb4c70bafaa2d05232705a, State: Initialized, Role: FOLLOWER
I20260619 16:46:25.628875 15264 consensus_queue.cc:260] T 700c272e6d53403ab4bbee7c4b1a630c P f2689ba3f5fb4c70bafaa2d05232705a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9dc2ad605434cb49fac1258d87806f5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37367 } } peers { permanent_uuid: "f2689ba3f5fb4c70bafaa2d05232705a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37273 } } peers { permanent_uuid: "09f7fa28cb44464fa809ee1bcf1a2c03" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42613 } }
I20260619 16:46:25.629946 15248 heartbeater.cc:503] Master 127.0.97.62:45429 was elected leader, sending a full tablet report...
I20260619 16:46:25.631858 15264 ts_tablet_manager.cc:1428] T 700c272e6d53403ab4bbee7c4b1a630c P f2689ba3f5fb4c70bafaa2d05232705a: Time spent starting tablet: real 0.014s user 0.010s sys 0.002s
I20260619 16:46:25.635262 15266 raft_consensus.cc:348] T 700c272e6d53403ab4bbee7c4b1a630c P 09f7fa28cb44464fa809ee1bcf1a2c03 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9dc2ad605434cb49fac1258d87806f5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37367 } } peers { permanent_uuid: "f2689ba3f5fb4c70bafaa2d05232705a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37273 } } peers { permanent_uuid: "09f7fa28cb44464fa809ee1bcf1a2c03" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42613 } }
I20260619 16:46:25.635603 15266 raft_consensus.cc:374] T 700c272e6d53403ab4bbee7c4b1a630c P 09f7fa28cb44464fa809ee1bcf1a2c03 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:46:25.635690 15266 raft_consensus.cc:729] T 700c272e6d53403ab4bbee7c4b1a630c P 09f7fa28cb44464fa809ee1bcf1a2c03 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09f7fa28cb44464fa809ee1bcf1a2c03, State: Initialized, Role: FOLLOWER
I20260619 16:46:25.635977 15265 raft_consensus.cc:348] T 700c272e6d53403ab4bbee7c4b1a630c P a9dc2ad605434cb49fac1258d87806f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9dc2ad605434cb49fac1258d87806f5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37367 } } peers { permanent_uuid: "f2689ba3f5fb4c70bafaa2d05232705a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37273 } } peers { permanent_uuid: "09f7fa28cb44464fa809ee1bcf1a2c03" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42613 } }
I20260619 16:46:25.636344 15265 raft_consensus.cc:374] T 700c272e6d53403ab4bbee7c4b1a630c P a9dc2ad605434cb49fac1258d87806f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:46:25.636245 15266 consensus_queue.cc:260] T 700c272e6d53403ab4bbee7c4b1a630c P 09f7fa28cb44464fa809ee1bcf1a2c03 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9dc2ad605434cb49fac1258d87806f5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37367 } } peers { permanent_uuid: "f2689ba3f5fb4c70bafaa2d05232705a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37273 } } peers { permanent_uuid: "09f7fa28cb44464fa809ee1bcf1a2c03" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42613 } }
I20260619 16:46:25.636425 15265 raft_consensus.cc:729] T 700c272e6d53403ab4bbee7c4b1a630c P a9dc2ad605434cb49fac1258d87806f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a9dc2ad605434cb49fac1258d87806f5, State: Initialized, Role: FOLLOWER
I20260619 16:46:25.636904 15265 consensus_queue.cc:260] T 700c272e6d53403ab4bbee7c4b1a630c P a9dc2ad605434cb49fac1258d87806f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9dc2ad605434cb49fac1258d87806f5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37367 } } peers { permanent_uuid: "f2689ba3f5fb4c70bafaa2d05232705a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37273 } } peers { permanent_uuid: "09f7fa28cb44464fa809ee1bcf1a2c03" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42613 } }
I20260619 16:46:25.637370 15117 heartbeater.cc:503] Master 127.0.97.62:45429 was elected leader, sending a full tablet report...
I20260619 16:46:25.638026 15266 ts_tablet_manager.cc:1428] T 700c272e6d53403ab4bbee7c4b1a630c P 09f7fa28cb44464fa809ee1bcf1a2c03: Time spent starting tablet: real 0.011s user 0.012s sys 0.000s
I20260619 16:46:25.638615 15265 ts_tablet_manager.cc:1428] T 700c272e6d53403ab4bbee7c4b1a630c P a9dc2ad605434cb49fac1258d87806f5: Time spent starting tablet: real 0.012s user 0.012s sys 0.000s
I20260619 16:46:25.697260 15270 raft_consensus.cc:482] T 700c272e6d53403ab4bbee7c4b1a630c P f2689ba3f5fb4c70bafaa2d05232705a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 16:46:25.697589 15270 raft_consensus.cc:504] T 700c272e6d53403ab4bbee7c4b1a630c P f2689ba3f5fb4c70bafaa2d05232705a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9dc2ad605434cb49fac1258d87806f5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37367 } } peers { permanent_uuid: "f2689ba3f5fb4c70bafaa2d05232705a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37273 } } peers { permanent_uuid: "09f7fa28cb44464fa809ee1bcf1a2c03" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42613 } }
I20260619 16:46:25.699298 15270 leader_election.cc:288] T 700c272e6d53403ab4bbee7c4b1a630c P f2689ba3f5fb4c70bafaa2d05232705a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a9dc2ad605434cb49fac1258d87806f5 (127.0.97.1:37367), 09f7fa28cb44464fa809ee1bcf1a2c03 (127.0.97.2:42613)
I20260619 16:46:25.705539 15262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:25.699709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:59109 (local address 127.0.97.2:42613)
0619 16:46:25.699898 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:25.699903 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:25.699988 (+ 85us) server_negotiation.cc:407] Connection header received
0619 16:46:25.700209 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:25.700215 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:25.700291 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:25.700406 (+ 115us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:46:25.700811 (+ 405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:25.701820 (+ 1009us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:25.703753 (+ 1933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:25.704718 (+ 965us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:25.704834 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:25.704924 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0619 16:46:25.705167 (+ 243us) server_negotiation.cc:299] Negotiation successful
0619 16:46:25.705285 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260619 16:46:25.705552 15250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:25.699585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:42613 (local address 127.0.97.3:59109)
0619 16:46:25.699886 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:25.699906 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 16:46:25.700069 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:46:25.700431 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:46:25.700436 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:46:25.700466 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:46:25.700653 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:25.700664 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:25.701944 (+ 1280us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:25.701950 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:25.703601 (+ 1651us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:25.703613 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:25.704862 (+ 1249us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:25.704868 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:25.705029 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:25.705053 (+ 24us) client_negotiation.cc:769] Sending connection context
0619 16:46:25.705131 (+ 78us) client_negotiation.cc:240] Negotiation successful
0619 16:46:25.705256 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"mutex_wait_us":2}
I20260619 16:46:25.706094 15258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:25.699709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:48303 (local address 127.0.97.1:37367)
0619 16:46:25.699903 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:25.699908 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:25.700579 (+ 671us) server_negotiation.cc:407] Connection header received
0619 16:46:25.700711 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:25.700716 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:25.700794 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:25.700866 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:46:25.701529 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:25.702502 (+ 973us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:25.704299 (+ 1797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:25.705170 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:25.705299 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:25.705396 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0619 16:46:25.705712 (+ 316us) server_negotiation.cc:299] Negotiation successful
0619 16:46:25.705864 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260619 16:46:25.706717 15273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:25.699678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:37367 (local address 127.0.97.3:48303)
0619 16:46:25.700466 (+ 788us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:25.700488 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0619 16:46:25.700606 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:46:25.700946 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:46:25.700951 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:46:25.700977 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:46:25.701344 (+ 367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:25.701356 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:25.702647 (+ 1291us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:25.702651 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:25.704124 (+ 1473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:25.704139 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:25.705387 (+ 1248us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:25.705394 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:25.705543 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:25.705565 (+ 22us) client_negotiation.cc:769] Sending connection context
0619 16:46:25.706271 (+ 706us) client_negotiation.cc:240] Negotiation successful
0619 16:46:25.706413 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":576,"thread_start_us":369,"threads_started":1}
I20260619 16:46:25.706919 15072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "700c272e6d53403ab4bbee7c4b1a630c" candidate_uuid: "f2689ba3f5fb4c70bafaa2d05232705a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09f7fa28cb44464fa809ee1bcf1a2c03" is_pre_election: true
I20260619 16:46:25.707485 15072 raft_consensus.cc:2454] T 700c272e6d53403ab4bbee7c4b1a630c P 09f7fa28cb44464fa809ee1bcf1a2c03 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f2689ba3f5fb4c70bafaa2d05232705a in term 0.
I20260619 16:46:25.707589 14940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "700c272e6d53403ab4bbee7c4b1a630c" candidate_uuid: "f2689ba3f5fb4c70bafaa2d05232705a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a9dc2ad605434cb49fac1258d87806f5" is_pre_election: true
I20260619 16:46:25.708040 14940 raft_consensus.cc:2454] T 700c272e6d53403ab4bbee7c4b1a630c P a9dc2ad605434cb49fac1258d87806f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f2689ba3f5fb4c70bafaa2d05232705a in term 0.
I20260619 16:46:25.708057 15137 leader_election.cc:302] T 700c272e6d53403ab4bbee7c4b1a630c P f2689ba3f5fb4c70bafaa2d05232705a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09f7fa28cb44464fa809ee1bcf1a2c03, f2689ba3f5fb4c70bafaa2d05232705a; no voters:
I20260619 16:46:25.708391 15270 raft_consensus.cc:2781] T 700c272e6d53403ab4bbee7c4b1a630c P f2689ba3f5fb4c70bafaa2d05232705a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 16:46:25.708547 15270 raft_consensus.cc:482] T 700c272e6d53403ab4bbee7c4b1a630c P f2689ba3f5fb4c70bafaa2d05232705a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 16:46:25.708654 15270 raft_consensus.cc:3037] T 700c272e6d53403ab4bbee7c4b1a630c P f2689ba3f5fb4c70bafaa2d05232705a [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:46:25.710878 15270 raft_consensus.cc:504] T 700c272e6d53403ab4bbee7c4b1a630c P f2689ba3f5fb4c70bafaa2d05232705a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9dc2ad605434cb49fac1258d87806f5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37367 } } peers { permanent_uuid: "f2689ba3f5fb4c70bafaa2d05232705a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37273 } } peers { permanent_uuid: "09f7fa28cb44464fa809ee1bcf1a2c03" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42613 } }
I20260619 16:46:25.711469 15270 leader_election.cc:288] T 700c272e6d53403ab4bbee7c4b1a630c P f2689ba3f5fb4c70bafaa2d05232705a [CANDIDATE]: Term 1 election: Requested vote from peers a9dc2ad605434cb49fac1258d87806f5 (127.0.97.1:37367), 09f7fa28cb44464fa809ee1bcf1a2c03 (127.0.97.2:42613)
I20260619 16:46:25.711923 15072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "700c272e6d53403ab4bbee7c4b1a630c" candidate_uuid: "f2689ba3f5fb4c70bafaa2d05232705a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09f7fa28cb44464fa809ee1bcf1a2c03"
I20260619 16:46:25.711915 14940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "700c272e6d53403ab4bbee7c4b1a630c" candidate_uuid: "f2689ba3f5fb4c70bafaa2d05232705a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a9dc2ad605434cb49fac1258d87806f5"
I20260619 16:46:25.712122 14940 raft_consensus.cc:3037] T 700c272e6d53403ab4bbee7c4b1a630c P a9dc2ad605434cb49fac1258d87806f5 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:46:25.712122 15072 raft_consensus.cc:3037] T 700c272e6d53403ab4bbee7c4b1a630c P 09f7fa28cb44464fa809ee1bcf1a2c03 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:46:25.715225 14940 raft_consensus.cc:2454] T 700c272e6d53403ab4bbee7c4b1a630c P a9dc2ad605434cb49fac1258d87806f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f2689ba3f5fb4c70bafaa2d05232705a in term 1.
I20260619 16:46:25.715225 15072 raft_consensus.cc:2454] T 700c272e6d53403ab4bbee7c4b1a630c P 09f7fa28cb44464fa809ee1bcf1a2c03 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f2689ba3f5fb4c70bafaa2d05232705a in term 1.
I20260619 16:46:25.715659 15139 leader_election.cc:302] T 700c272e6d53403ab4bbee7c4b1a630c P f2689ba3f5fb4c70bafaa2d05232705a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a9dc2ad605434cb49fac1258d87806f5, f2689ba3f5fb4c70bafaa2d05232705a; no voters:
I20260619 16:46:25.715947 15270 raft_consensus.cc:2781] T 700c272e6d53403ab4bbee7c4b1a630c P f2689ba3f5fb4c70bafaa2d05232705a [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:46:25.716368 15270 raft_consensus.cc:686] T 700c272e6d53403ab4bbee7c4b1a630c P f2689ba3f5fb4c70bafaa2d05232705a [term 1 LEADER]: Becoming Leader. State: Replica: f2689ba3f5fb4c70bafaa2d05232705a, State: Running, Role: LEADER
I20260619 16:46:25.716862 15270 consensus_queue.cc:237] T 700c272e6d53403ab4bbee7c4b1a630c P f2689ba3f5fb4c70bafaa2d05232705a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9dc2ad605434cb49fac1258d87806f5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37367 } } peers { permanent_uuid: "f2689ba3f5fb4c70bafaa2d05232705a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37273 } } peers { permanent_uuid: "09f7fa28cb44464fa809ee1bcf1a2c03" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42613 } }
I20260619 16:46:25.721230 14800 catalog_manager.cc:5495] T 700c272e6d53403ab4bbee7c4b1a630c P f2689ba3f5fb4c70bafaa2d05232705a reported cstate change: term changed from 0 to 1, leader changed from <none> to f2689ba3f5fb4c70bafaa2d05232705a (127.0.97.3). New cstate: current_term: 1 leader_uuid: "f2689ba3f5fb4c70bafaa2d05232705a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9dc2ad605434cb49fac1258d87806f5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37367 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f2689ba3f5fb4c70bafaa2d05232705a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37273 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "09f7fa28cb44464fa809ee1bcf1a2c03" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42613 } health_report { overall_health: UNKNOWN } } }
W20260619 16:46:25.745286 15249 tablet.cc:2369] T 700c272e6d53403ab4bbee7c4b1a630c P f2689ba3f5fb4c70bafaa2d05232705a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 16:46:25.747597 15263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:25.742665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43562 (local address 127.0.97.3:37273)
0619 16:46:25.742840 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:25.742844 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:25.742901 (+ 57us) server_negotiation.cc:407] Connection header received
0619 16:46:25.743059 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:25.743062 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:25.743135 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:25.743251 (+ 116us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 16:46:25.743599 (+ 348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:25.744282 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:25.745633 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:25.746198 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:25.746289 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:25.746678 (+ 389us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 16:46:25.747165 (+ 487us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 16:46:25.747266 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0619 16:46:25.747351 (+ 85us) server_negotiation.cc:299] Negotiation successful
0619 16:46:25.747411 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260619 16:46:25.755228 14940 raft_consensus.cc:1261] T 700c272e6d53403ab4bbee7c4b1a630c P a9dc2ad605434cb49fac1258d87806f5 [term 1 FOLLOWER]: Refusing update from remote peer f2689ba3f5fb4c70bafaa2d05232705a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:46:25.755228 15072 raft_consensus.cc:1261] T 700c272e6d53403ab4bbee7c4b1a630c P 09f7fa28cb44464fa809ee1bcf1a2c03 [term 1 FOLLOWER]: Refusing update from remote peer f2689ba3f5fb4c70bafaa2d05232705a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:46:25.756246 15274 consensus_queue.cc:1035] T 700c272e6d53403ab4bbee7c4b1a630c P f2689ba3f5fb4c70bafaa2d05232705a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a9dc2ad605434cb49fac1258d87806f5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37367 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:46:25.756654 15270 consensus_queue.cc:1035] T 700c272e6d53403ab4bbee7c4b1a630c P f2689ba3f5fb4c70bafaa2d05232705a [LEADER]: Connected to new peer: Peer: permanent_uuid: "09f7fa28cb44464fa809ee1bcf1a2c03" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42613 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:46:25.781586 14800 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:55774:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:46:25.781872 14800 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:55774:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:46:25.785392 14800 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e1e294d313564ec99776f3b6b38335d5: Sending DeleteTablet for 3 replicas of tablet 700c272e6d53403ab4bbee7c4b1a630c
I20260619 16:46:25.786525 14920 tablet_service.cc:1514] Processing DeleteTablet for tablet 700c272e6d53403ab4bbee7c4b1a630c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:46:25 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:36750
I20260619 16:46:25.786547 15183 tablet_service.cc:1514] Processing DeleteTablet for tablet 700c272e6d53403ab4bbee7c4b1a630c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:46:25 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:43556
I20260619 16:46:25.787205 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu pid 14837
I20260619 16:46:25.787205 15052 tablet_service.cc:1514] Processing DeleteTablet for tablet 700c272e6d53403ab4bbee7c4b1a630c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:46:25 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:39716
I20260619 16:46:25.807651 15258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:25.788762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36754 (local address 127.0.97.1:37367)
0619 16:46:25.788945 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:25.788950 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:25.788967 (+ 17us) server_negotiation.cc:407] Connection header received
0619 16:46:25.789024 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:25.789029 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:25.789107 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:25.789194 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:46:25.791111 (+ 1917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:25.792298 (+ 1187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:25.794300 (+ 2002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:25.794991 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:25.795105 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:25.798883 (+ 3778us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:46:25.798906 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:46:25.798929 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:46:25.798985 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:46:25.803205 (+ 4220us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:25.804274 (+ 1069us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:25.804286 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:25.804290 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:25.804455 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:25.804958 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:25.804964 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:25.804967 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:25.805470 (+ 503us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:46:25.805789 (+ 319us) server_negotiation.cc:1035] Waiting for connection context
0619 16:46:25.807007 (+ 1218us) server_negotiation.cc:299] Negotiation successful
0619 16:46:25.807244 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260619 16:46:25.808475 14980 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:46:25.809062 15288 tablet_replica.cc:330] T 700c272e6d53403ab4bbee7c4b1a630c P 09f7fa28cb44464fa809ee1bcf1a2c03: stopping tablet replica
I20260619 16:46:25.809576 15288 raft_consensus.cc:2229] T 700c272e6d53403ab4bbee7c4b1a630c P 09f7fa28cb44464fa809ee1bcf1a2c03 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 16:46:25.810143 15288 raft_consensus.cc:2258] T 700c272e6d53403ab4bbee7c4b1a630c P 09f7fa28cb44464fa809ee1bcf1a2c03 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:46:25.811066 15286 tablet_replica.cc:330] T 700c272e6d53403ab4bbee7c4b1a630c P f2689ba3f5fb4c70bafaa2d05232705a: stopping tablet replica
I20260619 16:46:25.811455 15286 raft_consensus.cc:2229] T 700c272e6d53403ab4bbee7c4b1a630c P f2689ba3f5fb4c70bafaa2d05232705a [term 1 LEADER]: Raft consensus shutting down.
I20260619 16:46:25.812019 15286 raft_consensus.cc:2258] T 700c272e6d53403ab4bbee7c4b1a630c P f2689ba3f5fb4c70bafaa2d05232705a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:46:25.813675 15288 ts_tablet_manager.cc:1905] T 700c272e6d53403ab4bbee7c4b1a630c P 09f7fa28cb44464fa809ee1bcf1a2c03: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:46:25.814159 15286 ts_tablet_manager.cc:1905] T 700c272e6d53403ab4bbee7c4b1a630c P f2689ba3f5fb4c70bafaa2d05232705a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:46:25.820298 15288 ts_tablet_manager.cc:1918] T 700c272e6d53403ab4bbee7c4b1a630c P 09f7fa28cb44464fa809ee1bcf1a2c03: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 16:46:25.820451 15288 log.cc:1196] T 700c272e6d53403ab4bbee7c4b1a630c P 09f7fa28cb44464fa809ee1bcf1a2c03: Deleting WAL directory at /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887385903558-388-0/minicluster-data/ts-1/wal/wals/700c272e6d53403ab4bbee7c4b1a630c
I20260619 16:46:25.820987 15288 ts_tablet_manager.cc:1939] T 700c272e6d53403ab4bbee7c4b1a630c P 09f7fa28cb44464fa809ee1bcf1a2c03: Deleting consensus metadata
I20260619 16:46:25.821133 15286 ts_tablet_manager.cc:1918] T 700c272e6d53403ab4bbee7c4b1a630c P f2689ba3f5fb4c70bafaa2d05232705a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 16:46:25.821324 15286 log.cc:1196] T 700c272e6d53403ab4bbee7c4b1a630c P f2689ba3f5fb4c70bafaa2d05232705a: Deleting WAL directory at /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887385903558-388-0/minicluster-data/ts-2/wal/wals/700c272e6d53403ab4bbee7c4b1a630c
I20260619 16:46:25.821902 15286 ts_tablet_manager.cc:1939] T 700c272e6d53403ab4bbee7c4b1a630c P f2689ba3f5fb4c70bafaa2d05232705a: Deleting consensus metadata
I20260619 16:46:25.822935 14785 catalog_manager.cc:4841] TS 09f7fa28cb44464fa809ee1bcf1a2c03 (127.0.97.2:42613): tablet 700c272e6d53403ab4bbee7c4b1a630c (table test-table [id=bbf7ff42f11c4f439078d9a9d22f6d18]) successfully deleted
I20260619 16:46:25.823453 14784 catalog_manager.cc:4841] TS f2689ba3f5fb4c70bafaa2d05232705a (127.0.97.3:37273): tablet 700c272e6d53403ab4bbee7c4b1a630c (table test-table [id=bbf7ff42f11c4f439078d9a9d22f6d18]) successfully deleted
I20260619 16:46:25.924726 15287 tablet_replica.cc:330] T 700c272e6d53403ab4bbee7c4b1a630c P a9dc2ad605434cb49fac1258d87806f5: stopping tablet replica
I20260619 16:46:25.925194 15287 raft_consensus.cc:2229] T 700c272e6d53403ab4bbee7c4b1a630c P a9dc2ad605434cb49fac1258d87806f5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 16:46:25.925709 15287 raft_consensus.cc:2258] T 700c272e6d53403ab4bbee7c4b1a630c P a9dc2ad605434cb49fac1258d87806f5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:46:25.926127 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu with pid 14837
W20260619 16:46:25.941016 14787 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.97.1:37367 (error 108)
I20260619 16:46:25.941220 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu pid 14989
I20260619 16:46:25.943120 15259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:25.942632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:37367 (local address 127.0.0.1:36756)
0619 16:46:25.942825 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:25.942943 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:37367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
W20260619 16:46:25.943713 14787 catalog_manager.cc:4584] TS a9dc2ad605434cb49fac1258d87806f5 (127.0.97.1:37367): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 700c272e6d53403ab4bbee7c4b1a630c: Network error: Client connection negotiation failed: client connection to 127.0.97.1:37367: connect: Connection refused (error 111)
I20260619 16:46:25.959167 15262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:25.942350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39726 (local address 127.0.97.2:42613)
0619 16:46:25.942542 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:25.942548 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:25.942566 (+ 18us) server_negotiation.cc:407] Connection header received
0619 16:46:25.942628 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:25.942633 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:25.942712 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:25.942804 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:46:25.944285 (+ 1481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:25.945006 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:25.946475 (+ 1469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:25.946864 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:25.946971 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:25.953635 (+ 6664us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:46:25.953651 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:46:25.953665 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:46:25.953712 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:46:25.956486 (+ 2774us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:25.957198 (+ 712us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:25.957205 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:25.957208 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:25.957326 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:25.957670 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:25.957673 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:25.957676 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:25.958075 (+ 399us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:46:25.958335 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0619 16:46:25.958646 (+ 311us) server_negotiation.cc:299] Negotiation successful
0619 16:46:25.958853 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260619 16:46:25.959825 15112 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:46:25.997486 15259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:25.996921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:37367 (local address 127.0.0.1:36766)
0619 16:46:25.997171 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:25.997286 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:37367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
I20260619 16:46:25.998658 15259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:25.998266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:37367 (local address 127.0.0.1:36774)
0619 16:46:25.998412 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:25.998499 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:37367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260619 16:46:26.047999 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu with pid 14989
I20260619 16:46:26.059088 15259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:26.058645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:37367 (local address 127.0.0.1:36786)
0619 16:46:26.058850 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:26.058932 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:37367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260619 16:46:26.060050 15259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:26.059723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:37367 (local address 127.0.0.1:36800)
0619 16:46:26.059864 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:26.059945 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:37367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260619 16:46:26.067921 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu pid 15120
I20260619 16:46:26.081414 15263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:26.068582 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43564 (local address 127.0.97.3:37273)
0619 16:46:26.068749 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:26.068758 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:26.068832 (+ 74us) server_negotiation.cc:407] Connection header received
0619 16:46:26.069008 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:26.069012 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:26.069074 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:26.069155 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:46:26.070614 (+ 1459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:26.071425 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:26.072685 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:26.073168 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:26.073278 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:26.075742 (+ 2464us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:46:26.075753 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:46:26.075770 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:46:26.075802 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:46:26.078820 (+ 3018us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:26.079494 (+ 674us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:26.079502 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:26.079504 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:26.079642 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:26.079982 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:26.079987 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:26.079989 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:26.080303 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:46:26.080596 (+ 293us) server_negotiation.cc:1035] Waiting for connection context
0619 16:46:26.080851 (+ 255us) server_negotiation.cc:299] Negotiation successful
0619 16:46:26.081057 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260619 16:46:26.082019 15243 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:46:26.140414 15259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:26.139976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:37367 (local address 127.0.0.1:36806)
0619 16:46:26.140169 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:26.140256 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:37367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260619 16:46:26.141497 15259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:26.141096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:37367 (local address 127.0.0.1:36814)
0619 16:46:26.141285 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:26.141395 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:37367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102}
I20260619 16:46:26.201128 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu with pid 15120
I20260619 16:46:26.215818 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu pid 14767
I20260619 16:46:26.216881 14830 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:46:26.325766 15259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:26.325244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:37367 (local address 127.0.0.1:36818)
0619 16:46:26.325439 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:26.325546 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:37367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260619 16:46:26.325774 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu with pid 14767
I20260619 16:46:26.343152 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu
/tmp/dist-test-taskEs2EH8/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887385903558-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887385903558-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887385903558-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887385903558-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45429
--webserver_interface=127.0.97.62
--webserver_port=34959
--keytab_file=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887385903558-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38239
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:45429
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887385903558-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887385903558-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887385903558-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:46:26.648790 15294 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:46:26.649065 15294 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:46:26.649178 15294 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:46:26.656234 15294 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:46:26.656337 15294 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:46:26.656384 15294 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:46:26.656461 15294 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:46:26.656515 15294 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:46:26.666061 15294 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38239
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887385903558-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887385903558-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:45429
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887385903558-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45429
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887385903558-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=34959
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887385903558-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:46:26.667306 15294 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:46:26.668540 15294 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:46:26.679425 15299 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:46:26.679505 15300 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:46:26.679468 15302 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:46:26.680337 15294 server_base.cc:1034] running on GCE node
I20260619 16:46:26.681052 15294 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:46:26.682423 15294 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:46:26.683617 15294 hybrid_clock.cc:648] HybridClock initialized: now 1781887586683574 us; error 38 us; skew 500 ppm
I20260619 16:46:26.683925 15294 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:46:26 dist-test-slave-wvz7 krb5kdc[14210](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887586, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:46:26.686921 15294 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260619 16:46:26.688342 15294 webserver.cc:460] Webserver started at http://127.0.97.62:34959/ using document root <none> and password file <none>
I20260619 16:46:26.688997 15294 fs_manager.cc:359] Metadata directory not provided
I20260619 16:46:26.689111 15294 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:46:26.694418 15294 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260619 16:46:26.697080 15309 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:46:26.698422 15294 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260619 16:46:26.698581 15294 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887385903558-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887385903558-388-0/minicluster-data/master-0/wal
uuid: "e1e294d313564ec99776f3b6b38335d5"
format_stamp: "Formatted at 2026-06-19 16:46:20 on dist-test-slave-wvz7"
server_key: "fc92fef4748c44f2be23bd7ef93fe4f1"
server_key_iv: "529772319276df9c93bee0a3ed2a668d"
server_key_version: "encryptionkey@0"
I20260619 16:46:26.699514 15294 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887385903558-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887385903558-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887385903558-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:46:26.710475 15294 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:46:26.714327 15294 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:46:26.714574 15294 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:46:26.731645 15294 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:45429
I20260619 16:46:26.731691 15360 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:45429 every 8 connection(s)
I20260619 16:46:26.733143 15294 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887385903558-388-0/minicluster-data/master-0/data/info.pb
I20260619 16:46:26.740051 15361 sys_catalog.cc:263] Verifying existing consensus state
I20260619 16:46:26.742398 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu as pid 15294
I20260619 16:46:26.743180 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu
/tmp/dist-test-taskEs2EH8/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887385903558-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887385903558-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887385903558-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887385903558-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:37367
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:45429
--webserver_port=42735
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887385903558-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38239
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887385903558-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887385903558-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887385903558-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 16:46:26.743412 15361 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e1e294d313564ec99776f3b6b38335d5: Bootstrap starting.
I20260619 16:46:26.768719 15361 log.cc:824] T 00000000000000000000000000000000 P e1e294d313564ec99776f3b6b38335d5: Log is configured to *not* fsync() on all Append() calls
I20260619 16:46:26.788784 15361 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e1e294d313564ec99776f3b6b38335d5: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 16:46:26.789405 15361 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e1e294d313564ec99776f3b6b38335d5: Bootstrap complete.
I20260619 16:46:26.802578 15361 raft_consensus.cc:348] T 00000000000000000000000000000000 P e1e294d313564ec99776f3b6b38335d5 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1e294d313564ec99776f3b6b38335d5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45429 } }
I20260619 16:46:26.803869 15361 raft_consensus.cc:729] T 00000000000000000000000000000000 P e1e294d313564ec99776f3b6b38335d5 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1e294d313564ec99776f3b6b38335d5, State: Initialized, Role: FOLLOWER
I20260619 16:46:26.804587 15361 consensus_queue.cc:260] T 00000000000000000000000000000000 P e1e294d313564ec99776f3b6b38335d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1e294d313564ec99776f3b6b38335d5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45429 } }
I20260619 16:46:26.804903 15361 raft_consensus.cc:388] T 00000000000000000000000000000000 P e1e294d313564ec99776f3b6b38335d5 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:46:26.805033 15361 raft_consensus.cc:482] T 00000000000000000000000000000000 P e1e294d313564ec99776f3b6b38335d5 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:46:26.805220 15361 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e1e294d313564ec99776f3b6b38335d5 [term 2 FOLLOWER]: Advancing to term 3
I20260619 16:46:26.811106 15361 raft_consensus.cc:504] T 00000000000000000000000000000000 P e1e294d313564ec99776f3b6b38335d5 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1e294d313564ec99776f3b6b38335d5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45429 } }
I20260619 16:46:26.811676 15361 leader_election.cc:302] T 00000000000000000000000000000000 P e1e294d313564ec99776f3b6b38335d5 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e1e294d313564ec99776f3b6b38335d5; no voters:
I20260619 16:46:26.812299 15361 leader_election.cc:288] T 00000000000000000000000000000000 P e1e294d313564ec99776f3b6b38335d5 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260619 16:46:26.812427 15365 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e1e294d313564ec99776f3b6b38335d5 [term 3 FOLLOWER]: Leader election won for term 3
I20260619 16:46:26.813421 15365 raft_consensus.cc:686] T 00000000000000000000000000000000 P e1e294d313564ec99776f3b6b38335d5 [term 3 LEADER]: Becoming Leader. State: Replica: e1e294d313564ec99776f3b6b38335d5, State: Running, Role: LEADER
I20260619 16:46:26.813911 15365 consensus_queue.cc:237] T 00000000000000000000000000000000 P e1e294d313564ec99776f3b6b38335d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1e294d313564ec99776f3b6b38335d5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45429 } }
I20260619 16:46:26.814304 15361 sys_catalog.cc:564] T 00000000000000000000000000000000 P e1e294d313564ec99776f3b6b38335d5 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:46:26.816075 15366 sys_catalog.cc:455] T 00000000000000000000000000000000 P e1e294d313564ec99776f3b6b38335d5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "e1e294d313564ec99776f3b6b38335d5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1e294d313564ec99776f3b6b38335d5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45429 } } }
I20260619 16:46:26.816393 15366 sys_catalog.cc:458] T 00000000000000000000000000000000 P e1e294d313564ec99776f3b6b38335d5 [sys.catalog]: This master's current role is: LEADER
I20260619 16:46:26.816632 15367 sys_catalog.cc:455] T 00000000000000000000000000000000 P e1e294d313564ec99776f3b6b38335d5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e1e294d313564ec99776f3b6b38335d5. Latest consensus state: current_term: 3 leader_uuid: "e1e294d313564ec99776f3b6b38335d5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1e294d313564ec99776f3b6b38335d5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45429 } } }
I20260619 16:46:26.817266 15367 sys_catalog.cc:458] T 00000000000000000000000000000000 P e1e294d313564ec99776f3b6b38335d5 [sys.catalog]: This master's current role is: LEADER
I20260619 16:46:26.821703 15370 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:46:26.830019 15370 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:46:26.831689 15370 catalog_manager.cc:1259] Loaded cluster ID: b3db8dcb2ff3459f9a5b16ab9e88649d
I20260619 16:46:26.831849 15370 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260619 16:46:26.833889 15370 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f90ee3c99ad google::LogMessage::Fail() at ??:0
@ 0x7f90ee3cc88a google::LogMessage::SendToLog() at ??:0
@ 0x7f90ee3c94de google::LogMessage::Flush() at ??:0
@ 0x7f90ee3caa19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f9103dfc512 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7f9103df9976 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f9103e81610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7f9103e81439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f90f1015345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f90f101891c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f90f1018799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f90f0ff3eaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7f90f42cd6db start_thread at ??:0
@ 0x7f90ed26361f clone at ??:0
W20260619 16:46:27.108987 15363 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:46:27.109275 15363 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:46:27.109455 15363 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:46:27.117067 15363 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:46:27.117192 15363 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:46:27.117348 15363 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260619 16:46:27.127146 15363 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38239
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887385903558-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887385903558-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887385903558-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:37367
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887385903558-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=42735
--tserver_master_addrs=127.0.97.62:45429
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887385903558-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:46:27.128468 15363 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:46:27.129859 15363 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:46:27.142755 15402 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:46:27.142935 15403 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:46:27.142849 15405 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:46:27.143893 15363 server_base.cc:1034] running on GCE node
I20260619 16:46:27.144500 15363 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:46:27.145409 15363 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:46:27.146612 15363 hybrid_clock.cc:648] HybridClock initialized: now 1781887587146493 us; error 108 us; skew 500 ppm
I20260619 16:46:27.146921 15363 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:46:27 dist-test-slave-wvz7 krb5kdc[14210](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887587, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:46:27.149936 15363 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260619 16:46:27.151341 15363 webserver.cc:460] Webserver started at http://127.0.97.1:42735/ using document root <none> and password file <none>
I20260619 16:46:27.151988 15363 fs_manager.cc:359] Metadata directory not provided
I20260619 16:46:27.152086 15363 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:46:27.157495 15363 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.000s
I20260619 16:46:27.160485 15412 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:46:27.161928 15363 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260619 16:46:27.162091 15363 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887385903558-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887385903558-388-0/minicluster-data/ts-0/wal
uuid: "a9dc2ad605434cb49fac1258d87806f5"
format_stamp: "Formatted at 2026-06-19 16:46:21 on dist-test-slave-wvz7"
server_key: "a1fdf62f258763c823e087b9c5d87f23"
server_key_iv: "8a28126bc939a78919f716c3739ca8da"
server_key_version: "encryptionkey@0"
I20260619 16:46:27.163131 15363 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887385903558-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887385903558-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887385903558-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:46:27.206561 15363 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:46:27.210739 15363 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:46:27.210994 15363 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:46:27.211977 15363 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:46:27.213848 15419 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20260619 16:46:27.219269 15363 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260619 16:46:27.219415 15363 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.001s sys 0.000s
I20260619 16:46:27.219542 15363 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260619 16:46:27.222579 15363 ts_tablet_manager.cc:610] Registered 1 tablets
I20260619 16:46:27.222661 15363 ts_tablet_manager.cc:589] Time spent register tablets: real 0.003s user 0.003s sys 0.000s
I20260619 16:46:27.222944 15419 tablet_bootstrap.cc:492] T 700c272e6d53403ab4bbee7c4b1a630c P a9dc2ad605434cb49fac1258d87806f5: Bootstrap starting.
I20260619 16:46:27.247097 15419 log.cc:824] T 700c272e6d53403ab4bbee7c4b1a630c P a9dc2ad605434cb49fac1258d87806f5: Log is configured to *not* fsync() on all Append() calls
I20260619 16:46:27.260488 15363 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:37367
I20260619 16:46:27.260512 15526 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:37367 every 8 connection(s)
I20260619 16:46:27.262077 15363 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887385903558-388-0/minicluster-data/ts-0/data/info.pb
I20260619 16:46:27.262730 15419 tablet_bootstrap.cc:492] T 700c272e6d53403ab4bbee7c4b1a630c P a9dc2ad605434cb49fac1258d87806f5: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 16:46:27.263453 15419 tablet_bootstrap.cc:492] T 700c272e6d53403ab4bbee7c4b1a630c P a9dc2ad605434cb49fac1258d87806f5: Bootstrap complete.
I20260619 16:46:27.264637 15419 ts_tablet_manager.cc:1397] T 700c272e6d53403ab4bbee7c4b1a630c P a9dc2ad605434cb49fac1258d87806f5: Time spent bootstrapping tablet: real 0.042s user 0.036s sys 0.004s
I20260619 16:46:27.267071 15529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:27.264911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45429 (local address 127.0.97.1:43249)
0619 16:46:27.265780 (+ 869us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:27.266032 (+ 252us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":564,"thread_start_us":380,"threads_started":1}
W20260619 16:46:27.267863 15527 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:45429 (0 consecutive failures): Network error: Failed to ping master at 127.0.97.62:45429: Client connection negotiation failed: client connection to 127.0.97.62:45429: connect: Connection refused (error 111)
I20260619 16:46:27.268694 15529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:27.268366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45429 (local address 127.0.97.1:49609)
0619 16:46:27.268516 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:27.268589 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260619 16:46:27.269526 15529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:27.269190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45429 (local address 127.0.97.1:47831)
0619 16:46:27.269323 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:27.269404 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260619 16:46:27.269771 15527 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260619 16:46:27.270327 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu as pid 15363
I20260619 16:46:27.271078 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu
/tmp/dist-test-taskEs2EH8/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887385903558-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887385903558-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887385903558-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887385903558-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:42613
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:45429
--webserver_port=35761
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887385903558-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38239
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887385903558-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887385903558-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887385903558-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 16:46:27.272523 15419 raft_consensus.cc:348] T 700c272e6d53403ab4bbee7c4b1a630c P a9dc2ad605434cb49fac1258d87806f5 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9dc2ad605434cb49fac1258d87806f5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37367 } } peers { permanent_uuid: "f2689ba3f5fb4c70bafaa2d05232705a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37273 } } peers { permanent_uuid: "09f7fa28cb44464fa809ee1bcf1a2c03" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42613 } }
I20260619 16:46:27.273658 15419 raft_consensus.cc:729] T 700c272e6d53403ab4bbee7c4b1a630c P a9dc2ad605434cb49fac1258d87806f5 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a9dc2ad605434cb49fac1258d87806f5, State: Initialized, Role: FOLLOWER
I20260619 16:46:27.274330 15419 consensus_queue.cc:260] T 700c272e6d53403ab4bbee7c4b1a630c P a9dc2ad605434cb49fac1258d87806f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9dc2ad605434cb49fac1258d87806f5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37367 } } peers { permanent_uuid: "f2689ba3f5fb4c70bafaa2d05232705a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37273 } } peers { permanent_uuid: "09f7fa28cb44464fa809ee1bcf1a2c03" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42613 } }
I20260619 16:46:27.281323 15419 ts_tablet_manager.cc:1428] T 700c272e6d53403ab4bbee7c4b1a630c P a9dc2ad605434cb49fac1258d87806f5: Time spent starting tablet: real 0.017s user 0.009s sys 0.008s
I20260619 16:46:27.281585 15529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:27.281163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45429 (local address 127.0.97.1:58313)
0619 16:46:27.281342 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:27.281433 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260619 16:46:27.282416 15529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:27.282132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45429 (local address 127.0.97.1:53033)
0619 16:46:27.282247 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:27.282309 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260619 16:46:27.589628 15531 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:46:27.589965 15531 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:46:27.590097 15531 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:46:27.597368 15531 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:46:27.597517 15531 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:46:27.597680 15531 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260619 16:46:27.607488 15531 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38239
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887385903558-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887385903558-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887385903558-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:42613
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887385903558-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=35761
--tserver_master_addrs=127.0.97.62:45429
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887385903558-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:46:27.608795 15531 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:46:27.610106 15531 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:46:27.622651 15537 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:46:27.622684 15536 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:46:27.622614 15539 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:46:27.623627 15531 server_base.cc:1034] running on GCE node
I20260619 16:46:27.624205 15531 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:46:27.625102 15531 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:46:27.626286 15531 hybrid_clock.cc:648] HybridClock initialized: now 1781887587626234 us; error 43 us; skew 500 ppm
I20260619 16:46:27.626579 15531 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:46:27 dist-test-slave-wvz7 krb5kdc[14210](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887587, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:46:27.629452 15531 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260619 16:46:27.630884 15531 webserver.cc:460] Webserver started at http://127.0.97.2:35761/ using document root <none> and password file <none>
I20260619 16:46:27.631577 15531 fs_manager.cc:359] Metadata directory not provided
I20260619 16:46:27.631695 15531 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:46:27.636921 15531 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 16:46:27.639575 15546 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:46:27.640933 15531 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 16:46:27.641098 15531 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887385903558-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887385903558-388-0/minicluster-data/ts-1/wal
uuid: "09f7fa28cb44464fa809ee1bcf1a2c03"
format_stamp: "Formatted at 2026-06-19 16:46:21 on dist-test-slave-wvz7"
server_key: "f18fa1c04ddfbf586c6a8778a22869b0"
server_key_iv: "7dfdcd84e637eff4f21842baa7227bf2"
server_key_version: "encryptionkey@0"
I20260619 16:46:27.642122 15531 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887385903558-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887385903558-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887385903558-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:46:27.660739 15531 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:46:27.664889 15531 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:46:27.665139 15531 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:46:27.666141 15531 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:46:27.667477 15531 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:46:27.667589 15531 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:46:27.667694 15531 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:46:27.667765 15531 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:46:27.696856 15531 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:42613
I20260619 16:46:27.696920 15658 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:42613 every 8 connection(s)
I20260619 16:46:27.698320 15531 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887385903558-388-0/minicluster-data/ts-1/data/info.pb
I20260619 16:46:27.702867 15661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:27.701097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45429 (local address 127.0.97.2:46159)
0619 16:46:27.701794 (+ 697us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:27.701992 (+ 198us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":428,"thread_start_us":268,"threads_started":1}
W20260619 16:46:27.703474 15659 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:45429 (0 consecutive failures): Network error: Failed to ping master at 127.0.97.62:45429: Client connection negotiation failed: client connection to 127.0.97.62:45429: connect: Connection refused (error 111)
I20260619 16:46:27.704193 15661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:27.703896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45429 (local address 127.0.97.2:41643)
0619 16:46:27.704029 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:27.704093 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260619 16:46:27.704975 15661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:27.704630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45429 (local address 127.0.97.2:57367)
0619 16:46:27.704777 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:27.704869 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260619 16:46:27.705181 15659 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260619 16:46:27.706118 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu as pid 15531
I20260619 16:46:27.706668 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu
/tmp/dist-test-taskEs2EH8/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887385903558-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887385903558-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887385903558-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887385903558-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:37273
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:45429
--webserver_port=42761
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887385903558-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38239
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887385903558-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887385903558-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887385903558-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:46:28.017125 15662 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:46:28.017405 15662 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:46:28.017503 15662 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:46:28.024624 15662 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:46:28.024757 15662 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:46:28.024914 15662 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260619 16:46:28.034603 15662 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38239
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887385903558-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887385903558-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887385903558-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:37273
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887385903558-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=42761
--tserver_master_addrs=127.0.97.62:45429
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887385903558-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:46:28.035869 15662 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:46:28.037194 15662 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:46:28.049752 15670 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:46:28.049865 15668 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:46:28.049857 15667 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:46:28.050966 15662 server_base.cc:1034] running on GCE node
I20260619 16:46:28.051612 15662 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:46:28.052675 15662 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:46:28.053839 15662 hybrid_clock.cc:648] HybridClock initialized: now 1781887588053776 us; error 50 us; skew 500 ppm
I20260619 16:46:28.054159 15662 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:46:28 dist-test-slave-wvz7 krb5kdc[14210](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887588, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:46:28.057260 15662 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260619 16:46:28.058710 15662 webserver.cc:460] Webserver started at http://127.0.97.3:42761/ using document root <none> and password file <none>
I20260619 16:46:28.059360 15662 fs_manager.cc:359] Metadata directory not provided
I20260619 16:46:28.059484 15662 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:46:28.064728 15662 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.002s
I20260619 16:46:28.067492 15677 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:46:28.068797 15662 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 16:46:28.068949 15662 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887385903558-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887385903558-388-0/minicluster-data/ts-2/wal
uuid: "f2689ba3f5fb4c70bafaa2d05232705a"
format_stamp: "Formatted at 2026-06-19 16:46:22 on dist-test-slave-wvz7"
server_key: "f8f2beb316fac04fe57535d8da6ec7d5"
server_key_iv: "1142e95cb44a1e97d33f4af87f3b0bf4"
server_key_version: "encryptionkey@0"
I20260619 16:46:28.069929 15662 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887385903558-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887385903558-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887385903558-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:46:28.090978 15662 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:46:28.094849 15662 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:46:28.095086 15662 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:46:28.096123 15662 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:46:28.097515 15662 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:46:28.097610 15662 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:46:28.097725 15662 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:46:28.097826 15662 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260619 16:46:28.127916 15662 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:37273
I20260619 16:46:28.127998 15789 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:37273 every 8 connection(s)
I20260619 16:46:28.129854 15662 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887385903558-388-0/minicluster-data/ts-2/data/info.pb
I20260619 16:46:28.135078 15792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:28.133321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45429 (local address 127.0.97.3:41565)
0619 16:46:28.134029 (+ 708us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:28.134234 (+ 205us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":458,"thread_start_us":259,"threads_started":1}
W20260619 16:46:28.135708 15790 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:45429 (0 consecutive failures): Network error: Failed to ping master at 127.0.97.62:45429: Client connection negotiation failed: client connection to 127.0.97.62:45429: connect: Connection refused (error 111)
I20260619 16:46:28.136535 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu as pid 15662
I20260619 16:46:28.136566 15792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:28.136174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45429 (local address 127.0.97.3:42647)
0619 16:46:28.136322 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:28.136402 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260619 16:46:28.137483 15792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:28.137101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45429 (local address 127.0.97.3:53083)
0619 16:46:28.137253 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:28.137330 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260619 16:46:28.137684 15790 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260619 16:46:28.284164 15794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:28.283344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45429 (local address 127.0.97.1:50977)
0619 16:46:28.283847 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:28.283947 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":402,"thread_start_us":249,"threads_started":1}
I20260619 16:46:28.706624 15795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:28.705845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45429 (local address 127.0.97.2:45447)
0619 16:46:28.706305 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:28.706394 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":348,"thread_start_us":226,"threads_started":1}
I20260619 16:46:28.766826 15796 raft_consensus.cc:482] T 700c272e6d53403ab4bbee7c4b1a630c P a9dc2ad605434cb49fac1258d87806f5 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 16:46:28.767140 15796 raft_consensus.cc:504] T 700c272e6d53403ab4bbee7c4b1a630c P a9dc2ad605434cb49fac1258d87806f5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9dc2ad605434cb49fac1258d87806f5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37367 } } peers { permanent_uuid: "f2689ba3f5fb4c70bafaa2d05232705a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37273 } } peers { permanent_uuid: "09f7fa28cb44464fa809ee1bcf1a2c03" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42613 } }
I20260619 16:46:28.768755 15796 leader_election.cc:288] T 700c272e6d53403ab4bbee7c4b1a630c P a9dc2ad605434cb49fac1258d87806f5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f2689ba3f5fb4c70bafaa2d05232705a (127.0.97.3:37273), 09f7fa28cb44464fa809ee1bcf1a2c03 (127.0.97.2:42613)
Jun 19 16:46:28 dist-test-slave-wvz7 krb5kdc[14210](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887587, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
Jun 19 16:46:28 dist-test-slave-wvz7 krb5kdc[14210](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887587, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260619 16:46:28.794221 15794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:28.769036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:42613 (local address 127.0.97.1:60289)
0619 16:46:28.769193 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:28.769230 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0619 16:46:28.769794 (+ 564us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:46:28.773683 (+ 3889us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:46:28.773694 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:46:28.774612 (+ 918us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:46:28.775615 (+ 1003us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:28.775643 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:28.778082 (+ 2439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:28.778088 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:28.780346 (+ 2258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:28.780364 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:28.781820 (+ 1456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:28.781827 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:28.782194 (+ 367us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:28.783142 (+ 948us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:46:28.783181 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:46:28.786505 (+ 3324us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:46:28.790616 (+ 4111us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:46:28.790621 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:46:28.790625 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:46:28.791080 (+ 455us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:46:28.791514 (+ 434us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:46:28.791518 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:46:28.791521 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:46:28.791641 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:46:28.792492 (+ 851us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:46:28.792496 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:46:28.792886 (+ 390us) client_negotiation.cc:769] Sending connection context
0619 16:46:28.793218 (+ 332us) client_negotiation.cc:240] Negotiation successful
0619 16:46:28.793768 (+ 550us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51,"mutex_wait_us":306,"spinlock_wait_cycles":31232}
I20260619 16:46:28.794327 15797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:28.769307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:45467 (local address 127.0.97.3:37273)
0619 16:46:28.769849 (+ 542us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:28.769866 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:28.770555 (+ 689us) server_negotiation.cc:407] Connection header received
0619 16:46:28.772550 (+ 1995us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:28.772562 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:28.773080 (+ 518us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:28.773493 (+ 413us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:46:28.775869 (+ 2376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:28.777865 (+ 1996us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:28.780505 (+ 2640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:28.781578 (+ 1073us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:28.781800 (+ 222us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:28.786501 (+ 4701us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:46:28.786510 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:46:28.786526 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:46:28.786576 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:46:28.790133 (+ 3557us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:28.790975 (+ 842us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:28.790982 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:28.790987 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:28.791152 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:28.791555 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:28.791558 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:28.791560 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:28.792204 (+ 644us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:46:28.792407 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0619 16:46:28.793534 (+ 1127us) server_negotiation.cc:299] Negotiation successful
0619 16:46:28.793933 (+ 399us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":360,"thread_start_us":246,"threads_started":1}
I20260619 16:46:28.794327 15798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:28.769469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:60289 (local address 127.0.97.2:42613)
0619 16:46:28.770362 (+ 893us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:28.770377 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:28.770435 (+ 58us) server_negotiation.cc:407] Connection header received
0619 16:46:28.772565 (+ 2130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:28.772574 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:28.773080 (+ 506us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:28.773494 (+ 414us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:46:28.775868 (+ 2374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:28.777889 (+ 2021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:28.780505 (+ 2616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:28.781578 (+ 1073us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:28.781800 (+ 222us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:28.786784 (+ 4984us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:46:28.786794 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:46:28.786812 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:46:28.786862 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:46:28.790478 (+ 3616us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:28.791214 (+ 736us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:28.791222 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:28.791229 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:28.791391 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:28.791749 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:28.791753 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:28.791756 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:28.792336 (+ 580us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:46:28.792505 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0619 16:46:28.793493 (+ 988us) server_negotiation.cc:299] Negotiation successful
0619 16:46:28.793945 (+ 452us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":711,"thread_start_us":534,"threads_started":1}
I20260619 16:46:28.794276 15799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:28.769050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:37273 (local address 127.0.97.1:45467)
0619 16:46:28.770439 (+ 1389us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:28.770453 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 16:46:28.770554 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:46:28.774072 (+ 3518us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:46:28.774079 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:46:28.774749 (+ 670us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:46:28.775615 (+ 866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:28.775643 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:28.778085 (+ 2442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:28.778090 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:28.780344 (+ 2254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:28.780360 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:28.781831 (+ 1471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:28.781837 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:28.782188 (+ 351us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:28.783142 (+ 954us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:46:28.783184 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:46:28.786168 (+ 2984us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:46:28.790294 (+ 4126us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:46:28.790301 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:46:28.790314 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:46:28.790833 (+ 519us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:46:28.791252 (+ 419us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:46:28.791256 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:46:28.791258 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:46:28.791440 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:46:28.792400 (+ 960us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:46:28.792423 (+ 23us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:46:28.792886 (+ 463us) client_negotiation.cc:769] Sending connection context
0619 16:46:28.793218 (+ 332us) client_negotiation.cc:240] Negotiation successful
0619 16:46:28.793920 (+ 702us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1298,"mutex_wait_us":1293,"spinlock_wait_cycles":67072,"thread_start_us":415,"threads_started":1}
W20260619 16:46:28.797583 15414 leader_election.cc:341] T 700c272e6d53403ab4bbee7c4b1a630c P a9dc2ad605434cb49fac1258d87806f5 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 09f7fa28cb44464fa809ee1bcf1a2c03 (127.0.97.2:42613): Not found: Tablet not found: 700c272e6d53403ab4bbee7c4b1a630c
W20260619 16:46:28.797865 15413 leader_election.cc:341] T 700c272e6d53403ab4bbee7c4b1a630c P a9dc2ad605434cb49fac1258d87806f5 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f2689ba3f5fb4c70bafaa2d05232705a (127.0.97.3:37273): Not found: Tablet not found: 700c272e6d53403ab4bbee7c4b1a630c
I20260619 16:46:28.798005 15413 leader_election.cc:302] T 700c272e6d53403ab4bbee7c4b1a630c P a9dc2ad605434cb49fac1258d87806f5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a9dc2ad605434cb49fac1258d87806f5; no voters: 09f7fa28cb44464fa809ee1bcf1a2c03, f2689ba3f5fb4c70bafaa2d05232705a
I20260619 16:46:28.798429 15796 raft_consensus.cc:2726] T 700c272e6d53403ab4bbee7c4b1a630c P a9dc2ad605434cb49fac1258d87806f5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 16:46:29.138986 15800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:29.138189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45429 (local address 127.0.97.3:36847)
0619 16:46:29.138634 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:29.138753 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":335,"thread_start_us":193,"threads_started":1}
I20260619 16:46:29.285465 15799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:29.285036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45429 (local address 127.0.97.1:40849)
0619 16:46:29.285201 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:29.285293 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260619 16:46:29.708076 15801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:29.707400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45429 (local address 127.0.97.2:38107)
0619 16:46:29.707829 (+ 429us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:29.707925 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":336,"thread_start_us":229,"threads_started":1}
I20260619 16:46:30.140429 15802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:30.139774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45429 (local address 127.0.97.3:33129)
0619 16:46:30.140197 (+ 423us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:30.140275 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":324,"thread_start_us":235,"threads_started":1}
I20260619 16:46:30.287302 15803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:30.286478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45429 (local address 127.0.97.1:58797)
0619 16:46:30.287025 (+ 547us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:30.287111 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":413,"thread_start_us":225,"threads_started":1}
I20260619 16:46:30.709637 15804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:30.708905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45429 (local address 127.0.97.2:52327)
0619 16:46:30.709358 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:30.709442 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":353,"thread_start_us":243,"threads_started":1}
I20260619 16:46:31.038973 15805 raft_consensus.cc:482] T 700c272e6d53403ab4bbee7c4b1a630c P a9dc2ad605434cb49fac1258d87806f5 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 16:46:31.039165 15805 raft_consensus.cc:504] T 700c272e6d53403ab4bbee7c4b1a630c P a9dc2ad605434cb49fac1258d87806f5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9dc2ad605434cb49fac1258d87806f5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37367 } } peers { permanent_uuid: "f2689ba3f5fb4c70bafaa2d05232705a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37273 } } peers { permanent_uuid: "09f7fa28cb44464fa809ee1bcf1a2c03" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42613 } }
I20260619 16:46:31.039804 15805 leader_election.cc:288] T 700c272e6d53403ab4bbee7c4b1a630c P a9dc2ad605434cb49fac1258d87806f5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f2689ba3f5fb4c70bafaa2d05232705a (127.0.97.3:37273), 09f7fa28cb44464fa809ee1bcf1a2c03 (127.0.97.2:42613)
W20260619 16:46:31.040761 15414 leader_election.cc:341] T 700c272e6d53403ab4bbee7c4b1a630c P a9dc2ad605434cb49fac1258d87806f5 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 09f7fa28cb44464fa809ee1bcf1a2c03 (127.0.97.2:42613): Not found: Tablet not found: 700c272e6d53403ab4bbee7c4b1a630c
W20260619 16:46:31.040930 15413 leader_election.cc:341] T 700c272e6d53403ab4bbee7c4b1a630c P a9dc2ad605434cb49fac1258d87806f5 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f2689ba3f5fb4c70bafaa2d05232705a (127.0.97.3:37273): Not found: Tablet not found: 700c272e6d53403ab4bbee7c4b1a630c
I20260619 16:46:31.041018 15413 leader_election.cc:302] T 700c272e6d53403ab4bbee7c4b1a630c P a9dc2ad605434cb49fac1258d87806f5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a9dc2ad605434cb49fac1258d87806f5; no voters: 09f7fa28cb44464fa809ee1bcf1a2c03, f2689ba3f5fb4c70bafaa2d05232705a
I20260619 16:46:31.041242 15805 raft_consensus.cc:2726] T 700c272e6d53403ab4bbee7c4b1a630c P a9dc2ad605434cb49fac1258d87806f5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 16:46:31.142010 15806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:31.141301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45429 (local address 127.0.97.3:58139)
0619 16:46:31.141770 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:31.141851 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":333,"thread_start_us":226,"threads_started":1}
I20260619 16:46:31.288996 15807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:31.288260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45429 (local address 127.0.97.1:41089)
0619 16:46:31.288720 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:31.288814 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":354,"thread_start_us":237,"threads_started":1}
I20260619 16:46:31.711148 15808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:31.710474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45429 (local address 127.0.97.2:55147)
0619 16:46:31.710875 (+ 401us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:31.710949 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":306,"thread_start_us":192,"threads_started":1}
I20260619 16:46:32.143499 15809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:32.142848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45429 (local address 127.0.97.3:57931)
0619 16:46:32.143238 (+ 390us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:32.143318 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":297,"thread_start_us":189,"threads_started":1}
I20260619 16:46:32.290858 15810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:32.290137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45429 (local address 127.0.97.1:42573)
0619 16:46:32.290585 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:32.290666 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":350,"thread_start_us":239,"threads_started":1}
I20260619 16:46:32.712777 15811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:32.711989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45429 (local address 127.0.97.2:46139)
0619 16:46:32.712487 (+ 498us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:32.712589 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":381,"thread_start_us":259,"threads_started":1}
I20260619 16:46:33.145056 15812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:33.144369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45429 (local address 127.0.97.3:45291)
0619 16:46:33.144756 (+ 387us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:33.144864 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":298,"thread_start_us":202,"threads_started":1}
I20260619 16:46:33.292430 15813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:33.291697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45429 (local address 127.0.97.1:43035)
0619 16:46:33.292135 (+ 438us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:33.292243 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":320,"thread_start_us":214,"threads_started":1}
I20260619 16:46:33.714473 15814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:33.713670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45429 (local address 127.0.97.2:44533)
0619 16:46:33.714176 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:33.714259 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":396,"thread_start_us":278,"threads_started":1}
I20260619 16:46:33.998237 15815 raft_consensus.cc:482] T 700c272e6d53403ab4bbee7c4b1a630c P a9dc2ad605434cb49fac1258d87806f5 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 16:46:33.998457 15815 raft_consensus.cc:504] T 700c272e6d53403ab4bbee7c4b1a630c P a9dc2ad605434cb49fac1258d87806f5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9dc2ad605434cb49fac1258d87806f5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37367 } } peers { permanent_uuid: "f2689ba3f5fb4c70bafaa2d05232705a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37273 } } peers { permanent_uuid: "09f7fa28cb44464fa809ee1bcf1a2c03" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42613 } }
I20260619 16:46:33.999116 15815 leader_election.cc:288] T 700c272e6d53403ab4bbee7c4b1a630c P a9dc2ad605434cb49fac1258d87806f5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f2689ba3f5fb4c70bafaa2d05232705a (127.0.97.3:37273), 09f7fa28cb44464fa809ee1bcf1a2c03 (127.0.97.2:42613)
W20260619 16:46:34.000115 15413 leader_election.cc:341] T 700c272e6d53403ab4bbee7c4b1a630c P a9dc2ad605434cb49fac1258d87806f5 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f2689ba3f5fb4c70bafaa2d05232705a (127.0.97.3:37273): Not found: Tablet not found: 700c272e6d53403ab4bbee7c4b1a630c
W20260619 16:46:34.000294 15414 leader_election.cc:341] T 700c272e6d53403ab4bbee7c4b1a630c P a9dc2ad605434cb49fac1258d87806f5 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 09f7fa28cb44464fa809ee1bcf1a2c03 (127.0.97.2:42613): Not found: Tablet not found: 700c272e6d53403ab4bbee7c4b1a630c
I20260619 16:46:34.000411 15414 leader_election.cc:302] T 700c272e6d53403ab4bbee7c4b1a630c P a9dc2ad605434cb49fac1258d87806f5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a9dc2ad605434cb49fac1258d87806f5; no voters: 09f7fa28cb44464fa809ee1bcf1a2c03, f2689ba3f5fb4c70bafaa2d05232705a
I20260619 16:46:34.000699 15815 raft_consensus.cc:2726] T 700c272e6d53403ab4bbee7c4b1a630c P a9dc2ad605434cb49fac1258d87806f5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 16:46:34.146720 15816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:34.145982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45429 (local address 127.0.97.3:46157)
0619 16:46:34.146435 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:34.146541 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":345,"thread_start_us":227,"threads_started":1}
I20260619 16:46:34.294102 15817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:34.293290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45429 (local address 127.0.97.1:42435)
0619 16:46:34.293804 (+ 514us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:34.293903 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":377,"thread_start_us":262,"threads_started":1}
I20260619 16:46:34.716061 15818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:34.715259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45429 (local address 127.0.97.2:37713)
0619 16:46:34.715790 (+ 531us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:34.715872 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":425,"thread_start_us":261,"threads_started":1}
I20260619 16:46:35.148324 15819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:35.147550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45429 (local address 127.0.97.3:53951)
0619 16:46:35.148060 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:35.148143 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":387,"thread_start_us":223,"threads_started":1}
I20260619 16:46:35.295632 15820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:35.294964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45429 (local address 127.0.97.1:33375)
0619 16:46:35.295392 (+ 428us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:35.295470 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":331,"thread_start_us":224,"threads_started":1}
I20260619 16:46:35.717689 15821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:35.716933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45429 (local address 127.0.97.2:53067)
0619 16:46:35.717424 (+ 491us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:35.717512 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":351,"thread_start_us":248,"threads_started":1}
I20260619 16:46:36.149955 15822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:36.149207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45429 (local address 127.0.97.3:51461)
0619 16:46:36.149656 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:36.149772 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":350,"thread_start_us":215,"threads_started":1}
I20260619 16:46:36.297387 15823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:36.296614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45429 (local address 127.0.97.1:49757)
0619 16:46:36.297125 (+ 511us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:36.297203 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":383,"thread_start_us":249,"threads_started":1}
I20260619 16:46:36.719379 15824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:36.718566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45429 (local address 127.0.97.2:43775)
0619 16:46:36.719095 (+ 529us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:36.719197 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":409,"thread_start_us":265,"threads_started":1}
I20260619 16:46:37.151470 15825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:37.150746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45429 (local address 127.0.97.3:44881)
0619 16:46:37.151211 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:37.151289 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":370,"thread_start_us":223,"threads_started":1}
I20260619 16:46:37.299096 15826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:37.298377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45429 (local address 127.0.97.1:51641)
0619 16:46:37.298836 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:37.298933 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":358,"thread_start_us":233,"threads_started":1}
I20260619 16:46:37.720921 15827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:37.720277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45429 (local address 127.0.97.2:36831)
0619 16:46:37.720673 (+ 396us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:37.720751 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":302,"thread_start_us":207,"threads_started":1}
I20260619 16:46:37.941831 15828 raft_consensus.cc:482] T 700c272e6d53403ab4bbee7c4b1a630c P a9dc2ad605434cb49fac1258d87806f5 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 16:46:37.942030 15828 raft_consensus.cc:504] T 700c272e6d53403ab4bbee7c4b1a630c P a9dc2ad605434cb49fac1258d87806f5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9dc2ad605434cb49fac1258d87806f5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37367 } } peers { permanent_uuid: "f2689ba3f5fb4c70bafaa2d05232705a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37273 } } peers { permanent_uuid: "09f7fa28cb44464fa809ee1bcf1a2c03" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42613 } }
I20260619 16:46:37.942616 15828 leader_election.cc:288] T 700c272e6d53403ab4bbee7c4b1a630c P a9dc2ad605434cb49fac1258d87806f5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f2689ba3f5fb4c70bafaa2d05232705a (127.0.97.3:37273), 09f7fa28cb44464fa809ee1bcf1a2c03 (127.0.97.2:42613)
W20260619 16:46:37.943521 15414 leader_election.cc:341] T 700c272e6d53403ab4bbee7c4b1a630c P a9dc2ad605434cb49fac1258d87806f5 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 09f7fa28cb44464fa809ee1bcf1a2c03 (127.0.97.2:42613): Not found: Tablet not found: 700c272e6d53403ab4bbee7c4b1a630c
W20260619 16:46:37.943676 15413 leader_election.cc:341] T 700c272e6d53403ab4bbee7c4b1a630c P a9dc2ad605434cb49fac1258d87806f5 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f2689ba3f5fb4c70bafaa2d05232705a (127.0.97.3:37273): Not found: Tablet not found: 700c272e6d53403ab4bbee7c4b1a630c
I20260619 16:46:37.943828 15413 leader_election.cc:302] T 700c272e6d53403ab4bbee7c4b1a630c P a9dc2ad605434cb49fac1258d87806f5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a9dc2ad605434cb49fac1258d87806f5; no voters: 09f7fa28cb44464fa809ee1bcf1a2c03, f2689ba3f5fb4c70bafaa2d05232705a
I20260619 16:46:37.944082 15828 raft_consensus.cc:2726] T 700c272e6d53403ab4bbee7c4b1a630c P a9dc2ad605434cb49fac1258d87806f5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 16:46:38.153334 15829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:38.152566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45429 (local address 127.0.97.3:49701)
0619 16:46:38.153011 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:38.153111 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":321,"thread_start_us":221,"threads_started":1}
I20260619 16:46:38.300726 15830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:38.299974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45429 (local address 127.0.97.1:53119)
0619 16:46:38.300452 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:38.300536 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":377,"thread_start_us":239,"threads_started":1}
I20260619 16:46:38.722500 15831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:38.721804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45429 (local address 127.0.97.2:34081)
0619 16:46:38.722233 (+ 429us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:38.722308 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":325,"thread_start_us":203,"threads_started":1}
I20260619 16:46:39.154973 15832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:39.154237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45429 (local address 127.0.97.3:40407)
0619 16:46:39.154727 (+ 490us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:39.154807 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":400,"thread_start_us":252,"threads_started":1}
I20260619 16:46:39.302361 15833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:39.301586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45429 (local address 127.0.97.1:36197)
0619 16:46:39.302073 (+ 487us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:39.302154 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":375,"thread_start_us":277,"threads_started":1}
I20260619 16:46:39.724287 15834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:39.723537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45429 (local address 127.0.97.2:54923)
0619 16:46:39.724026 (+ 489us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:39.724109 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":395,"thread_start_us":251,"threads_started":1}
I20260619 16:46:40.156610 15835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:40.155885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45429 (local address 127.0.97.3:42493)
0619 16:46:40.156352 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:40.156439 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":351,"thread_start_us":233,"threads_started":1}
I20260619 16:46:40.304121 15836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:40.303247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45429 (local address 127.0.97.1:37821)
0619 16:46:40.303757 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:40.303873 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":387,"thread_start_us":271,"threads_started":1}
I20260619 16:46:40.726068 15837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:40.725204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45429 (local address 127.0.97.2:44249)
0619 16:46:40.725787 (+ 583us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:40.725887 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":426,"thread_start_us":285,"threads_started":1}
I20260619 16:46:41.158478 15838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:41.157590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45429 (local address 127.0.97.3:53221)
0619 16:46:41.158158 (+ 568us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:41.158265 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":442,"thread_start_us":293,"threads_started":1}
I20260619 16:46:41.305814 15839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:41.305061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45429 (local address 127.0.97.1:44519)
0619 16:46:41.305512 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:41.305595 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":341,"thread_start_us":216,"threads_started":1}
I20260619 16:46:41.500404 15840 raft_consensus.cc:482] T 700c272e6d53403ab4bbee7c4b1a630c P a9dc2ad605434cb49fac1258d87806f5 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 16:46:41.500641 15840 raft_consensus.cc:504] T 700c272e6d53403ab4bbee7c4b1a630c P a9dc2ad605434cb49fac1258d87806f5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9dc2ad605434cb49fac1258d87806f5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37367 } } peers { permanent_uuid: "f2689ba3f5fb4c70bafaa2d05232705a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37273 } } peers { permanent_uuid: "09f7fa28cb44464fa809ee1bcf1a2c03" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42613 } }
I20260619 16:46:41.501412 15840 leader_election.cc:288] T 700c272e6d53403ab4bbee7c4b1a630c P a9dc2ad605434cb49fac1258d87806f5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f2689ba3f5fb4c70bafaa2d05232705a (127.0.97.3:37273), 09f7fa28cb44464fa809ee1bcf1a2c03 (127.0.97.2:42613)
W20260619 16:46:41.502326 15413 leader_election.cc:341] T 700c272e6d53403ab4bbee7c4b1a630c P a9dc2ad605434cb49fac1258d87806f5 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f2689ba3f5fb4c70bafaa2d05232705a (127.0.97.3:37273): Not found: Tablet not found: 700c272e6d53403ab4bbee7c4b1a630c
W20260619 16:46:41.502506 15414 leader_election.cc:341] T 700c272e6d53403ab4bbee7c4b1a630c P a9dc2ad605434cb49fac1258d87806f5 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 09f7fa28cb44464fa809ee1bcf1a2c03 (127.0.97.2:42613): Not found: Tablet not found: 700c272e6d53403ab4bbee7c4b1a630c
I20260619 16:46:41.502628 15414 leader_election.cc:302] T 700c272e6d53403ab4bbee7c4b1a630c P a9dc2ad605434cb49fac1258d87806f5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a9dc2ad605434cb49fac1258d87806f5; no voters: 09f7fa28cb44464fa809ee1bcf1a2c03, f2689ba3f5fb4c70bafaa2d05232705a
I20260619 16:46:41.502878 15840 raft_consensus.cc:2726] T 700c272e6d53403ab4bbee7c4b1a630c P a9dc2ad605434cb49fac1258d87806f5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 16:46:41.727747 15841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:41.726991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45429 (local address 127.0.97.2:54249)
0619 16:46:41.727455 (+ 464us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:41.727542 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":367,"thread_start_us":246,"threads_started":1}
I20260619 16:46:42.160159 15842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:42.159358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45429 (local address 127.0.97.3:60739)
0619 16:46:42.159876 (+ 518us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:42.159967 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":415,"thread_start_us":281,"threads_started":1}
I20260619 16:46:42.307780 15843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:42.307072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45429 (local address 127.0.97.1:42065)
0619 16:46:42.307524 (+ 452us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:42.307611 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":350,"thread_start_us":234,"threads_started":1}
I20260619 16:46:42.729790 15844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:42.728922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45429 (local address 127.0.97.2:44805)
0619 16:46:42.729484 (+ 562us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:42.729587 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":435,"thread_start_us":312,"threads_started":1}
I20260619 16:46:43.146888 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu pid 15363
I20260619 16:46:43.161973 15846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:43.161228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45429 (local address 127.0.97.3:41385)
0619 16:46:43.161642 (+ 414us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:43.161768 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":294,"thread_start_us":201,"threads_started":1}
I20260619 16:46:43.166617 15845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:43.147867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60768 (local address 127.0.97.1:37367)
0619 16:46:43.148437 (+ 570us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:43.148448 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:43.148479 (+ 31us) server_negotiation.cc:407] Connection header received
0619 16:46:43.148551 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:43.148560 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:43.148786 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:43.148933 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:46:43.150741 (+ 1808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:43.152202 (+ 1461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:43.153873 (+ 1671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:43.154487 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:43.154601 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:43.161689 (+ 7088us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:46:43.161703 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:46:43.161757 (+ 54us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:46:43.161797 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:46:43.164348 (+ 2551us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:43.164935 (+ 587us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:43.164941 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:43.164943 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:43.165061 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:43.165327 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:43.165330 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:43.165332 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:43.165640 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:46:43.165902 (+ 262us) server_negotiation.cc:1035] Waiting for connection context
0619 16:46:43.166120 (+ 218us) server_negotiation.cc:299] Negotiation successful
0619 16:46:43.166305 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":380,"thread_start_us":222,"threads_started":1}
I20260619 16:46:43.167768 15522 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:46:43.259075 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu with pid 15363
I20260619 16:46:43.273056 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu pid 15531
I20260619 16:46:43.285600 15848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:43.273887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43808 (local address 127.0.97.2:42613)
0619 16:46:43.274354 (+ 467us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:43.274359 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:43.274374 (+ 15us) server_negotiation.cc:407] Connection header received
0619 16:46:43.274424 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:43.274428 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:43.274485 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:43.274590 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:46:43.275862 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:43.276594 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:43.277802 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:43.278203 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:43.278320 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:43.280651 (+ 2331us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:46:43.280656 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:46:43.280659 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:46:43.280692 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:46:43.283155 (+ 2463us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:43.283767 (+ 612us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:43.283772 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:43.283774 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:43.283907 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:43.284229 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:43.284232 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:43.284235 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:43.284493 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:46:43.284735 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0619 16:46:43.284995 (+ 260us) server_negotiation.cc:299] Negotiation successful
0619 16:46:43.285202 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":344,"thread_start_us":228,"threads_started":1}
I20260619 16:46:43.286343 15654 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:46:43.367594 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu with pid 15531
I20260619 16:46:43.381604 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu pid 15662
I20260619 16:46:43.394546 15850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:43.382460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57644 (local address 127.0.97.3:37273)
0619 16:46:43.382985 (+ 525us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:43.382989 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:43.383006 (+ 17us) server_negotiation.cc:407] Connection header received
0619 16:46:43.383080 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:43.383084 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:43.383179 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:43.383274 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:46:43.384599 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:43.385396 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:43.386832 (+ 1436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:43.387253 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:43.387359 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:43.390024 (+ 2665us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:46:43.390030 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:46:43.390033 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:46:43.390067 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:46:43.392445 (+ 2378us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:43.393036 (+ 591us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:43.393039 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:43.393042 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:43.393158 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:43.393427 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:43.393430 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:43.393432 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:43.393678 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:46:43.393912 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0619 16:46:43.394114 (+ 202us) server_negotiation.cc:299] Negotiation successful
0619 16:46:43.394276 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":412,"thread_start_us":278,"threads_started":1}
I20260619 16:46:43.395205 15785 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:46:43.475852 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu with pid 15662
2026-06-19T16:46:43Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (24173 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887385903558-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 16:46:43 dist-test-slave-wvz7 krb5kdc[15854](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 16:46:43 dist-test-slave-wvz7 krb5kdc[15854](info): set up 2 sockets
Jun 19 16:46:43 dist-test-slave-wvz7 krb5kdc[15854](info): commencing operation
krb5kdc: starting...
W20260619 16:46:44.486313 388 mini_kdc.cc:121] Time spent starting KDC: real 0.975s user 0.008s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887385903558-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887385903558-388-0/krb5kdc/test-user.keytab.
Jun 19 16:46:44 dist-test-slave-wvz7 krb5kdc[15854](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887604, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T16:46:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T16:46:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887385903558-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887385903558-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260619 16:46:44.703176 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu
/tmp/dist-test-taskEs2EH8/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887385903558-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887385903558-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887385903558-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887385903558-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:44977
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887385903558-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45131
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:44977
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887385903558-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887385903558-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887385903558-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:46:45.030474 15867 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:46:45.030755 15867 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:46:45.030840 15867 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:46:45.037916 15867 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260619 16:46:45.038009 15867 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260619 16:46:45.038044 15867 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260619 16:46:45.038095 15867 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:46:45.038146 15867 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:46:45.038210 15867 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:46:45.038252 15867 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:46:45.038309 15867 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:46:45.047919 15867 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45131
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887385903558-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887385903558-388-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:44977
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887385903558-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:44977
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887385903558-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887385903558-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:46:45.049129 15867 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:46:45.050379 15867 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:46:45.061235 15872 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:46:45.061308 15875 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:46:45.061722 15867 server_base.cc:1034] running on GCE node
W20260619 16:46:45.061276 15873 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:46:45.062651 15867 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:46:45.063939 15867 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:46:45.065136 15867 hybrid_clock.cc:648] HybridClock initialized: now 1781887605065078 us; error 38 us; skew 500 ppm
I20260619 16:46:45.065466 15867 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:46:45 dist-test-slave-wvz7 krb5kdc[15854](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887605, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:46:45.068341 15867 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260619 16:46:45.069669 15867 webserver.cc:460] Webserver started at http://127.0.97.62:43343/ using document root <none> and password file <none>
I20260619 16:46:45.070333 15867 fs_manager.cc:359] Metadata directory not provided
I20260619 16:46:45.070427 15867 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:46:45.070704 15867 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:46:45.073450 15867 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887385903558-388-0/minicluster-data/master-0/data/instance:
uuid: "460891f6ca034e7c99713cb56b72d30f"
format_stamp: "Formatted at 2026-06-19 16:46:45 on dist-test-slave-wvz7"
server_key: "45c073fb68fba1d7d41be8c7cb093973"
server_key_iv: "5428244b0fcbb52060183807b9da9d45"
server_key_version: "encryptionkey@0"
I20260619 16:46:45.074180 15867 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887385903558-388-0/minicluster-data/master-0/wal/instance:
uuid: "460891f6ca034e7c99713cb56b72d30f"
format_stamp: "Formatted at 2026-06-19 16:46:45 on dist-test-slave-wvz7"
server_key: "45c073fb68fba1d7d41be8c7cb093973"
server_key_iv: "5428244b0fcbb52060183807b9da9d45"
server_key_version: "encryptionkey@0"
I20260619 16:46:45.078843 15867 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260619 16:46:45.081943 15882 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:46:45.083134 15867 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 16:46:45.083303 15867 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887385903558-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887385903558-388-0/minicluster-data/master-0/wal
uuid: "460891f6ca034e7c99713cb56b72d30f"
format_stamp: "Formatted at 2026-06-19 16:46:45 on dist-test-slave-wvz7"
server_key: "45c073fb68fba1d7d41be8c7cb093973"
server_key_iv: "5428244b0fcbb52060183807b9da9d45"
server_key_version: "encryptionkey@0"
I20260619 16:46:45.083487 15867 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887385903558-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887385903558-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887385903558-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:46:45.093566 15867 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:46:45.097082 15867 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:46:45.097316 15867 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:46:45.113670 15867 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:44977
I20260619 16:46:45.113691 15933 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:44977 every 8 connection(s)
I20260619 16:46:45.115073 15867 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887385903558-388-0/minicluster-data/master-0/data/info.pb
I20260619 16:46:45.118608 15934 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:46:45.123875 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu as pid 15867
I20260619 16:46:45.124094 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887385903558-388-0/minicluster-data/master-0/wal/instance
I20260619 16:46:45.124905 388 external_mini_cluster.cc:1428] Setting key 6fea59d142d18bfdfe31c2ede1231359
I20260619 16:46:45.129508 15934 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 460891f6ca034e7c99713cb56b72d30f: Bootstrap starting.
I20260619 16:46:45.132956 15934 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 460891f6ca034e7c99713cb56b72d30f: Neither blocks nor log segments found. Creating new log.
I20260619 16:46:45.134131 15934 log.cc:824] T 00000000000000000000000000000000 P 460891f6ca034e7c99713cb56b72d30f: Log is configured to *not* fsync() on all Append() calls
I20260619 16:46:45.137066 15934 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 460891f6ca034e7c99713cb56b72d30f: No bootstrap required, opened a new log
Jun 19 16:46:45 dist-test-slave-wvz7 krb5kdc[15854](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887604, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 16:46:45.146659 15934 raft_consensus.cc:348] T 00000000000000000000000000000000 P 460891f6ca034e7c99713cb56b72d30f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "460891f6ca034e7c99713cb56b72d30f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44977 } }
I20260619 16:46:45.147107 15934 raft_consensus.cc:374] T 00000000000000000000000000000000 P 460891f6ca034e7c99713cb56b72d30f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:46:45.147223 15934 raft_consensus.cc:729] T 00000000000000000000000000000000 P 460891f6ca034e7c99713cb56b72d30f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 460891f6ca034e7c99713cb56b72d30f, State: Initialized, Role: FOLLOWER
I20260619 16:46:45.147879 15934 consensus_queue.cc:260] T 00000000000000000000000000000000 P 460891f6ca034e7c99713cb56b72d30f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "460891f6ca034e7c99713cb56b72d30f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44977 } }
I20260619 16:46:45.148140 15934 raft_consensus.cc:388] T 00000000000000000000000000000000 P 460891f6ca034e7c99713cb56b72d30f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:46:45.148272 15934 raft_consensus.cc:482] T 00000000000000000000000000000000 P 460891f6ca034e7c99713cb56b72d30f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:46:45.148442 15934 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 460891f6ca034e7c99713cb56b72d30f [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:46:45.148658 15937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:45.127072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40964 (local address 127.0.97.62:44977)
0619 16:46:45.127859 (+ 787us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:45.127871 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:45.127910 (+ 39us) server_negotiation.cc:407] Connection header received
0619 16:46:45.129507 (+ 1597us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:45.129530 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:45.129963 (+ 433us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:45.130290 (+ 327us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:46:45.131972 (+ 1682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:45.133312 (+ 1340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:45.135058 (+ 1746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:45.135811 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:45.135936 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:45.140436 (+ 4500us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:46:45.140446 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:46:45.140468 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:46:45.140514 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:46:45.144287 (+ 3773us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:45.144909 (+ 622us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:45.144918 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:45.144927 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:45.145146 (+ 219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:45.145540 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:45.145545 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:45.145549 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:45.146234 (+ 685us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:46:45.146526 (+ 292us) server_negotiation.cc:1035] Waiting for connection context
0619 16:46:45.146916 (+ 390us) server_negotiation.cc:299] Negotiation successful
0619 16:46:45.147356 (+ 440us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":582,"thread_start_us":368,"threads_started":1}
I20260619 16:46:45.150772 15934 raft_consensus.cc:504] T 00000000000000000000000000000000 P 460891f6ca034e7c99713cb56b72d30f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "460891f6ca034e7c99713cb56b72d30f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44977 } }
I20260619 16:46:45.151219 15934 leader_election.cc:302] T 00000000000000000000000000000000 P 460891f6ca034e7c99713cb56b72d30f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 460891f6ca034e7c99713cb56b72d30f; no voters:
I20260619 16:46:45.151743 15934 leader_election.cc:288] T 00000000000000000000000000000000 P 460891f6ca034e7c99713cb56b72d30f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 16:46:45.151860 15939 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 460891f6ca034e7c99713cb56b72d30f [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:46:45.152518 15939 raft_consensus.cc:686] T 00000000000000000000000000000000 P 460891f6ca034e7c99713cb56b72d30f [term 1 LEADER]: Becoming Leader. State: Replica: 460891f6ca034e7c99713cb56b72d30f, State: Running, Role: LEADER
I20260619 16:46:45.153038 15939 consensus_queue.cc:237] T 00000000000000000000000000000000 P 460891f6ca034e7c99713cb56b72d30f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "460891f6ca034e7c99713cb56b72d30f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44977 } }
I20260619 16:46:45.153466 15934 sys_catalog.cc:564] T 00000000000000000000000000000000 P 460891f6ca034e7c99713cb56b72d30f [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:46:45.155671 15941 sys_catalog.cc:455] T 00000000000000000000000000000000 P 460891f6ca034e7c99713cb56b72d30f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 460891f6ca034e7c99713cb56b72d30f. Latest consensus state: current_term: 1 leader_uuid: "460891f6ca034e7c99713cb56b72d30f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "460891f6ca034e7c99713cb56b72d30f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44977 } } }
I20260619 16:46:45.156054 15941 sys_catalog.cc:458] T 00000000000000000000000000000000 P 460891f6ca034e7c99713cb56b72d30f [sys.catalog]: This master's current role is: LEADER
I20260619 16:46:45.156931 15946 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:46:45.156747 15940 sys_catalog.cc:455] T 00000000000000000000000000000000 P 460891f6ca034e7c99713cb56b72d30f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "460891f6ca034e7c99713cb56b72d30f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "460891f6ca034e7c99713cb56b72d30f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44977 } } }
I20260619 16:46:45.157097 15940 sys_catalog.cc:458] T 00000000000000000000000000000000 P 460891f6ca034e7c99713cb56b72d30f [sys.catalog]: This master's current role is: LEADER
I20260619 16:46:45.163280 15946 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:46:45.173942 15946 catalog_manager.cc:1347] Generated new cluster ID: 5075d572c7964b97b3497f85f2f1032f
I20260619 16:46:45.174173 15946 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:46:45.194432 15946 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 16:46:45.195735 15946 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:46:45.207376 15946 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 460891f6ca034e7c99713cb56b72d30f: Generated new TSK 0
I20260619 16:46:45.208195 15946 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887385903558-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887385903558-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260619 16:46:45.269557 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu
/tmp/dist-test-taskEs2EH8/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887385903558-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887385903558-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887385903558-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887385903558-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44977
--keytab_file=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887385903558-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45131
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887385903558-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887385903558-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887385903558-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:46:45.596166 15960 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:46:45.596431 15960 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:46:45.596520 15960 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:46:45.603811 15960 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:46:45.603922 15960 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:46:45.604036 15960 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260619 16:46:45.613626 15960 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45131
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887385903558-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887385903558-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887385903558-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887385903558-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44977
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887385903558-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:46:45.614867 15960 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:46:45.616075 15960 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:46:45.628715 15965 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:46:45.628639 15968 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:46:45.628679 15966 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:46:45.629835 15960 server_base.cc:1034] running on GCE node
I20260619 16:46:45.630394 15960 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:46:45.631313 15960 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:46:45.632513 15960 hybrid_clock.cc:648] HybridClock initialized: now 1781887605632456 us; error 51 us; skew 500 ppm
I20260619 16:46:45.632794 15960 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:46:45 dist-test-slave-wvz7 krb5kdc[15854](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887605, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:46:45.635818 15960 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260619 16:46:45.637262 15960 webserver.cc:460] Webserver started at http://127.0.97.1:43223/ using document root <none> and password file <none>
I20260619 16:46:45.638085 15960 fs_manager.cc:359] Metadata directory not provided
I20260619 16:46:45.638206 15960 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:46:45.638458 15960 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:46:45.641151 15960 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887385903558-388-0/minicluster-data/ts-0/data/instance:
uuid: "be15a6b9c4824e229834519ec2b988d0"
format_stamp: "Formatted at 2026-06-19 16:46:45 on dist-test-slave-wvz7"
server_key: "3c2545cdbeb192463e0a296228190a40"
server_key_iv: "8d196395b19860edfa33881c321b1719"
server_key_version: "encryptionkey@0"
I20260619 16:46:45.641885 15960 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887385903558-388-0/minicluster-data/ts-0/wal/instance:
uuid: "be15a6b9c4824e229834519ec2b988d0"
format_stamp: "Formatted at 2026-06-19 16:46:45 on dist-test-slave-wvz7"
server_key: "3c2545cdbeb192463e0a296228190a40"
server_key_iv: "8d196395b19860edfa33881c321b1719"
server_key_version: "encryptionkey@0"
I20260619 16:46:45.646689 15960 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260619 16:46:45.650017 15975 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:46:45.651305 15960 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260619 16:46:45.651439 15960 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887385903558-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887385903558-388-0/minicluster-data/ts-0/wal
uuid: "be15a6b9c4824e229834519ec2b988d0"
format_stamp: "Formatted at 2026-06-19 16:46:45 on dist-test-slave-wvz7"
server_key: "3c2545cdbeb192463e0a296228190a40"
server_key_iv: "8d196395b19860edfa33881c321b1719"
server_key_version: "encryptionkey@0"
I20260619 16:46:45.651599 15960 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887385903558-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887385903558-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887385903558-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:46:45.664656 15960 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:46:45.668579 15960 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:46:45.668815 15960 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:46:45.669633 15960 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:46:45.670872 15960 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:46:45.670943 15960 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:46:45.671030 15960 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:46:45.671073 15960 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:46:45.699276 15960 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:42315
I20260619 16:46:45.699316 16087 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:42315 every 8 connection(s)
I20260619 16:46:45.700508 15960 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887385903558-388-0/minicluster-data/ts-0/data/info.pb
I20260619 16:46:45.703397 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu as pid 15960
I20260619 16:46:45.703615 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887385903558-388-0/minicluster-data/ts-0/wal/instance
I20260619 16:46:45.704453 388 external_mini_cluster.cc:1428] Setting key 160f6fe7949bb86c142003480233206a
Jun 19 16:46:45 dist-test-slave-wvz7 krb5kdc[15854](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887605, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 16:46:45.722009 16090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:45.703030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:58503 (local address 127.0.97.62:44977)
0619 16:46:45.703458 (+ 428us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:45.703463 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:45.704525 (+ 1062us) server_negotiation.cc:407] Connection header received
0619 16:46:45.706738 (+ 2213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:45.706743 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:45.706823 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:45.706933 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:46:45.709054 (+ 2121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:45.710046 (+ 992us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:45.712307 (+ 2261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:45.712791 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:45.712885 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:45.716688 (+ 3803us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:46:45.716693 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:46:45.716697 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:46:45.716733 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:46:45.719177 (+ 2444us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:45.719846 (+ 669us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:45.719851 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:45.719853 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:45.719968 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:45.720355 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:45.720360 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:45.720364 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:45.720621 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:46:45.720754 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0619 16:46:45.721412 (+ 658us) server_negotiation.cc:299] Negotiation successful
0619 16:46:45.721640 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":325,"thread_start_us":223,"threads_started":1}
I20260619 16:46:45.722923 16091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:45.703640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44977 (local address 127.0.97.1:58503)
0619 16:46:45.704368 (+ 728us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:45.704410 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0619 16:46:45.706447 (+ 2037us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:46:45.707246 (+ 799us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:46:45.707260 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:46:45.708003 (+ 743us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:46:45.708864 (+ 861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:45.708890 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:45.710223 (+ 1333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:45.710229 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:45.712169 (+ 1940us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:45.712186 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:45.713009 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:45.713016 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:45.713308 (+ 292us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:45.714079 (+ 771us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:46:45.714113 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:46:45.716443 (+ 2330us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:46:45.719310 (+ 2867us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:46:45.719318 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:46:45.719334 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:46:45.719737 (+ 403us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:46:45.720065 (+ 328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:46:45.720070 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:46:45.720074 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:46:45.720256 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:46:45.720735 (+ 479us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:46:45.720742 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:46:45.721052 (+ 310us) client_negotiation.cc:769] Sending connection context
0619 16:46:45.721294 (+ 242us) client_negotiation.cc:240] Negotiation successful
0619 16:46:45.721829 (+ 535us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":499,"thread_start_us":297,"threads_started":1}
I20260619 16:46:45.724421 16088 heartbeater.cc:347] Connected to a master server at 127.0.97.62:44977
I20260619 16:46:45.724717 16088 heartbeater.cc:464] Registering TS with master...
I20260619 16:46:45.725369 16088 heartbeater.cc:511] Master 127.0.97.62:44977 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260619 16:46:45.727339 15899 ts_manager.cc:194] Registered new tserver with Master: be15a6b9c4824e229834519ec2b988d0 (127.0.97.1:42315)
I20260619 16:46:45.728680 15899 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:58503
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887385903558-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887385903558-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260619 16:46:45.753003 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu
/tmp/dist-test-taskEs2EH8/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887385903558-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887385903558-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887385903558-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887385903558-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44977
--keytab_file=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887385903558-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45131
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887385903558-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887385903558-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887385903558-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:46:46.066541 16094 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:46:46.066823 16094 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:46:46.066941 16094 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:46:46.074015 16094 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:46:46.074116 16094 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:46:46.074270 16094 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260619 16:46:46.083696 16094 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45131
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887385903558-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887385903558-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887385903558-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887385903558-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44977
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887385903558-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:46:46.084961 16094 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:46:46.086277 16094 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:46:46.098836 16099 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:46:46.098768 16102 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:46:46.098789 16100 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:46:46.099727 16094 server_base.cc:1034] running on GCE node
I20260619 16:46:46.100282 16094 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:46:46.101256 16094 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:46:46.102456 16094 hybrid_clock.cc:648] HybridClock initialized: now 1781887606102411 us; error 41 us; skew 500 ppm
I20260619 16:46:46.102726 16094 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:46:46 dist-test-slave-wvz7 krb5kdc[15854](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887606, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:46:46.105834 16094 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260619 16:46:46.107278 16094 webserver.cc:460] Webserver started at http://127.0.97.2:36797/ using document root <none> and password file <none>
I20260619 16:46:46.107934 16094 fs_manager.cc:359] Metadata directory not provided
I20260619 16:46:46.108049 16094 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:46:46.108336 16094 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:46:46.111205 16094 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887385903558-388-0/minicluster-data/ts-1/data/instance:
uuid: "8b323dc8ce0f412db10a79763cb324ce"
format_stamp: "Formatted at 2026-06-19 16:46:46 on dist-test-slave-wvz7"
server_key: "e57f6b3f703bb844abffa382690ab4e7"
server_key_iv: "ffb1bf17e0f01f46f7035afa583b8e56"
server_key_version: "encryptionkey@0"
I20260619 16:46:46.111914 16094 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887385903558-388-0/minicluster-data/ts-1/wal/instance:
uuid: "8b323dc8ce0f412db10a79763cb324ce"
format_stamp: "Formatted at 2026-06-19 16:46:46 on dist-test-slave-wvz7"
server_key: "e57f6b3f703bb844abffa382690ab4e7"
server_key_iv: "ffb1bf17e0f01f46f7035afa583b8e56"
server_key_version: "encryptionkey@0"
I20260619 16:46:46.118222 16094 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260619 16:46:46.122797 16109 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:46:46.124385 16094 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.005s sys 0.000s
I20260619 16:46:46.124544 16094 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887385903558-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887385903558-388-0/minicluster-data/ts-1/wal
uuid: "8b323dc8ce0f412db10a79763cb324ce"
format_stamp: "Formatted at 2026-06-19 16:46:46 on dist-test-slave-wvz7"
server_key: "e57f6b3f703bb844abffa382690ab4e7"
server_key_iv: "ffb1bf17e0f01f46f7035afa583b8e56"
server_key_version: "encryptionkey@0"
I20260619 16:46:46.124763 16094 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887385903558-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887385903558-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887385903558-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:46:46.150770 16094 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:46:46.154626 16094 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:46:46.154889 16094 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:46:46.155894 16094 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:46:46.157130 16094 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:46:46.157197 16094 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:46:46.157289 16094 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:46:46.157330 16094 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:46:46.186913 16094 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:34415
I20260619 16:46:46.186941 16221 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:34415 every 8 connection(s)
I20260619 16:46:46.188203 16094 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887385903558-388-0/minicluster-data/ts-1/data/info.pb
I20260619 16:46:46.193872 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu as pid 16094
I20260619 16:46:46.194074 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887385903558-388-0/minicluster-data/ts-1/wal/instance
I20260619 16:46:46.194507 388 external_mini_cluster.cc:1428] Setting key cf5541155a11926e81d589a843209ecd
Jun 19 16:46:46 dist-test-slave-wvz7 krb5kdc[15854](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887606, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 16:46:46.207902 16090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:46.190693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:38415 (local address 127.0.97.62:44977)
0619 16:46:46.190872 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:46.190877 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:46.192031 (+ 1154us) server_negotiation.cc:407] Connection header received
0619 16:46:46.193995 (+ 1964us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:46.194006 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:46.194099 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:46.194205 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:46:46.195834 (+ 1629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:46.196620 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:46.198173 (+ 1553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:46.198583 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:46.198691 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:46.202308 (+ 3617us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:46:46.202313 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:46:46.202317 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:46:46.202351 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:46:46.204859 (+ 2508us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:46.205581 (+ 722us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:46.205585 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:46.205588 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:46.205715 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:46.206151 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:46.206156 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:46.206158 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:46.206432 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:46:46.206562 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0619 16:46:46.207259 (+ 697us) server_negotiation.cc:299] Negotiation successful
0619 16:46:46.207504 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260619 16:46:46.208861 16224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:46.191131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44977 (local address 127.0.97.2:38415)
0619 16:46:46.191878 (+ 747us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:46.191915 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0619 16:46:46.193650 (+ 1735us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:46:46.194469 (+ 819us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:46:46.194477 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:46:46.195006 (+ 529us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:46:46.195673 (+ 667us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:46.195690 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:46.196759 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:46.196763 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:46.198041 (+ 1278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:46.198053 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:46.198787 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:46.198794 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:46.199012 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:46.199681 (+ 669us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:46:46.199706 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:46:46.202043 (+ 2337us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:46:46.205000 (+ 2957us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:46:46.205009 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:46:46.205028 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:46:46.205456 (+ 428us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:46:46.205841 (+ 385us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:46:46.205846 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:46:46.205850 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:46:46.206039 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:46:46.206557 (+ 518us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:46:46.206565 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:46:46.206901 (+ 336us) client_negotiation.cc:769] Sending connection context
0619 16:46:46.207140 (+ 239us) client_negotiation.cc:240] Negotiation successful
0619 16:46:46.207690 (+ 550us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":521,"thread_start_us":331,"threads_started":1}
I20260619 16:46:46.210443 16222 heartbeater.cc:347] Connected to a master server at 127.0.97.62:44977
I20260619 16:46:46.210755 16222 heartbeater.cc:464] Registering TS with master...
I20260619 16:46:46.211421 16222 heartbeater.cc:511] Master 127.0.97.62:44977 requested a full tablet report, sending...
I20260619 16:46:46.212972 15899 ts_manager.cc:194] Registered new tserver with Master: 8b323dc8ce0f412db10a79763cb324ce (127.0.97.2:34415)
I20260619 16:46:46.214088 15899 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:38415
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887385903558-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887385903558-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260619 16:46:46.244443 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu
/tmp/dist-test-taskEs2EH8/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887385903558-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887385903558-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887385903558-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887385903558-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44977
--keytab_file=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887385903558-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45131
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887385903558-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887385903558-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887385903558-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:46:46.549642 16227 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:46:46.549932 16227 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:46:46.550050 16227 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:46:46.557301 16227 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:46:46.557408 16227 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:46:46.557546 16227 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260619 16:46:46.567075 16227 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45131
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887385903558-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887385903558-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887385903558-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887385903558-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44977
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887385903558-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:46:46.568274 16227 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:46:46.569485 16227 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:46:46.581692 16232 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:46:46.581749 16233 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:46:46.581785 16235 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:46:46.582402 16227 server_base.cc:1034] running on GCE node
I20260619 16:46:46.582974 16227 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:46:46.583889 16227 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:46:46.585069 16227 hybrid_clock.cc:648] HybridClock initialized: now 1781887606585010 us; error 53 us; skew 500 ppm
I20260619 16:46:46.585351 16227 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:46:46 dist-test-slave-wvz7 krb5kdc[15854](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887606, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:46:46.588307 16227 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260619 16:46:46.589639 16227 webserver.cc:460] Webserver started at http://127.0.97.3:37241/ using document root <none> and password file <none>
I20260619 16:46:46.590288 16227 fs_manager.cc:359] Metadata directory not provided
I20260619 16:46:46.590399 16227 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:46:46.590673 16227 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:46:46.593443 16227 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887385903558-388-0/minicluster-data/ts-2/data/instance:
uuid: "1efaebd5eb7a4bba9df40e4b0093546c"
format_stamp: "Formatted at 2026-06-19 16:46:46 on dist-test-slave-wvz7"
server_key: "926aba1dd5bd0febc4ab69191b110e40"
server_key_iv: "236335f03fb312342966ceaf411779d1"
server_key_version: "encryptionkey@0"
I20260619 16:46:46.594254 16227 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887385903558-388-0/minicluster-data/ts-2/wal/instance:
uuid: "1efaebd5eb7a4bba9df40e4b0093546c"
format_stamp: "Formatted at 2026-06-19 16:46:46 on dist-test-slave-wvz7"
server_key: "926aba1dd5bd0febc4ab69191b110e40"
server_key_iv: "236335f03fb312342966ceaf411779d1"
server_key_version: "encryptionkey@0"
I20260619 16:46:46.599249 16227 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260619 16:46:46.602681 16242 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:46:46.604018 16227 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260619 16:46:46.604166 16227 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887385903558-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887385903558-388-0/minicluster-data/ts-2/wal
uuid: "1efaebd5eb7a4bba9df40e4b0093546c"
format_stamp: "Formatted at 2026-06-19 16:46:46 on dist-test-slave-wvz7"
server_key: "926aba1dd5bd0febc4ab69191b110e40"
server_key_iv: "236335f03fb312342966ceaf411779d1"
server_key_version: "encryptionkey@0"
I20260619 16:46:46.604339 16227 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887385903558-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887385903558-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887385903558-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:46:46.632754 16227 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:46:46.636545 16227 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:46:46.636804 16227 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:46:46.637717 16227 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:46:46.639000 16227 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:46:46.639073 16227 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:46:46.639191 16227 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:46:46.639242 16227 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:46:46.668237 16227 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:37569
I20260619 16:46:46.668283 16354 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:37569 every 8 connection(s)
I20260619 16:46:46.669397 16227 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887385903558-388-0/minicluster-data/ts-2/data/info.pb
I20260619 16:46:46.674887 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu as pid 16227
I20260619 16:46:46.675063 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887385903558-388-0/minicluster-data/ts-2/wal/instance
I20260619 16:46:46.675508 388 external_mini_cluster.cc:1428] Setting key b8409037ff9725c1ee814333313b246a
Jun 19 16:46:46 dist-test-slave-wvz7 krb5kdc[15854](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887606, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 16:46:46.688273 16090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:46.671711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:47371 (local address 127.0.97.62:44977)
0619 16:46:46.671875 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:46.671880 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:46.672992 (+ 1112us) server_negotiation.cc:407] Connection header received
0619 16:46:46.674858 (+ 1866us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:46.674863 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:46.674938 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:46.675021 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:46:46.676468 (+ 1447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:46.677386 (+ 918us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:46.679217 (+ 1831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:46.679611 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:46.679703 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:46.682873 (+ 3170us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:46:46.682878 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:46:46.682881 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:46:46.682914 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:46:46.685319 (+ 2405us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:46.686062 (+ 743us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:46.686066 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:46.686069 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:46.686219 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:46.686684 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:46.686688 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:46.686690 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:46.687008 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:46:46.687203 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0619 16:46:46.687798 (+ 595us) server_negotiation.cc:299] Negotiation successful
0619 16:46:46.687988 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260619 16:46:46.689271 16357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:46.672197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44977 (local address 127.0.97.3:47371)
0619 16:46:46.672829 (+ 632us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:46.672879 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0619 16:46:46.674597 (+ 1718us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:46:46.675252 (+ 655us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:46:46.675260 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:46:46.675774 (+ 514us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:46:46.676322 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:46.676344 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:46.677528 (+ 1184us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:46.677533 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:46.679088 (+ 1555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:46.679099 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:46.679785 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:46.679790 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:46.679971 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:46.680574 (+ 603us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:46:46.680594 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:46:46.682680 (+ 2086us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:46:46.685443 (+ 2763us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:46:46.685449 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:46:46.685460 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:46:46.685946 (+ 486us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:46:46.686336 (+ 390us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:46:46.686340 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:46:46.686342 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:46:46.686522 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:46:46.687165 (+ 643us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:46:46.687171 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:46:46.687467 (+ 296us) client_negotiation.cc:769] Sending connection context
0619 16:46:46.687687 (+ 220us) client_negotiation.cc:240] Negotiation successful
0619 16:46:46.688098 (+ 411us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":423,"thread_start_us":253,"threads_started":1}
I20260619 16:46:46.690572 16355 heartbeater.cc:347] Connected to a master server at 127.0.97.62:44977
I20260619 16:46:46.690882 16355 heartbeater.cc:464] Registering TS with master...
I20260619 16:46:46.691552 16355 heartbeater.cc:511] Master 127.0.97.62:44977 requested a full tablet report, sending...
I20260619 16:46:46.692863 15899 ts_manager.cc:194] Registered new tserver with Master: 1efaebd5eb7a4bba9df40e4b0093546c (127.0.97.3:37569)
I20260619 16:46:46.693620 15899 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:47371
I20260619 16:46:46.702910 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 16:46:46.718153 16090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:46.707011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40972 (local address 127.0.97.62:44977)
0619 16:46:46.707154 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:46.707160 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:46.707620 (+ 460us) server_negotiation.cc:407] Connection header received
0619 16:46:46.707753 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:46.707756 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:46.707811 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:46.707883 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:46:46.709077 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:46.709800 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:46.711099 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:46.711481 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:46.711566 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:46.713676 (+ 2110us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:46:46.713681 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:46:46.713684 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:46:46.713712 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:46:46.716061 (+ 2349us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:46.716659 (+ 598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:46.716662 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:46.716665 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:46.716772 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:46.717056 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:46.717059 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:46.717061 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:46.717337 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:46:46.717506 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0619 16:46:46.717781 (+ 275us) server_negotiation.cc:299] Negotiation successful
0619 16:46:46.717935 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260619 16:46:46.721619 15899 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40972:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 16:46:46.728295 15899 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 16:46:46.731453 16088 heartbeater.cc:503] Master 127.0.97.62:44977 was elected leader, sending a full tablet report...
I20260619 16:46:46.749466 16370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:46.740299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40578 (local address 127.0.97.1:42315)
0619 16:46:46.741410 (+ 1111us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:46.741420 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:46.742108 (+ 688us) server_negotiation.cc:407] Connection header received
0619 16:46:46.742215 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:46.742224 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:46.742482 (+ 258us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:46.742659 (+ 177us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:46:46.743163 (+ 504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:46.744178 (+ 1015us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:46.746784 (+ 2606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:46.748107 (+ 1323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:46.748212 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:46.748440 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0619 16:46:46.749052 (+ 612us) server_negotiation.cc:299] Negotiation successful
0619 16:46:46.749195 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":931,"thread_start_us":1234,"threads_started":1}
I20260619 16:46:46.750360 16369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:46.739976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42315 (local address 127.0.0.1:40578)
0619 16:46:46.741851 (+ 1875us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:46.741874 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0619 16:46:46.741993 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:46:46.742685 (+ 692us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:46:46.742691 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:46:46.742719 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:46:46.743013 (+ 294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:46.743023 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:46.744297 (+ 1274us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:46.744302 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:46.746622 (+ 2320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:46.746647 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:46.748583 (+ 1936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:46.748589 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:46.748789 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:46.748842 (+ 53us) client_negotiation.cc:769] Sending connection context
0619 16:46:46.749759 (+ 917us) client_negotiation.cc:240] Negotiation successful
0619 16:46:46.749948 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1720,"mutex_wait_us":42,"thread_start_us":1574,"threads_started":1}
I20260619 16:46:46.750519 16372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:46.741154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40428 (local address 127.0.97.2:34415)
0619 16:46:46.742115 (+ 961us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:46.742124 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:46.742590 (+ 466us) server_negotiation.cc:407] Connection header received
0619 16:46:46.742883 (+ 293us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:46.742892 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:46.743120 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:46.743267 (+ 147us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:46:46.744136 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:46.745795 (+ 1659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:46.747925 (+ 2130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:46.749353 (+ 1428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:46.749458 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:46.749621 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0619 16:46:46.750070 (+ 449us) server_negotiation.cc:299] Negotiation successful
0619 16:46:46.750223 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":824,"thread_start_us":702,"threads_started":1}
I20260619 16:46:46.750540 16371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:46.740862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:34415 (local address 127.0.0.1:40428)
0619 16:46:46.742479 (+ 1617us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:46.742501 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0619 16:46:46.742698 (+ 197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:46:46.743548 (+ 850us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:46:46.743553 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:46:46.743579 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:46:46.743949 (+ 370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:46.743959 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:46.745920 (+ 1961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:46.745924 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:46.747763 (+ 1839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:46.747777 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:46.749767 (+ 1990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:46.749773 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:46.749913 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:46.749936 (+ 23us) client_negotiation.cc:769] Sending connection context
0619 16:46:46.750013 (+ 77us) client_negotiation.cc:240] Negotiation successful
0619 16:46:46.750145 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1520,"mutex_wait_us":40,"thread_start_us":1395,"threads_started":1}
I20260619 16:46:46.751469 16367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:46.739575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:37569 (local address 127.0.0.1:49612)
0619 16:46:46.740454 (+ 879us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:46.740503 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0619 16:46:46.740654 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:46:46.745252 (+ 4598us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:46:46.745256 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:46:46.745281 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:46:46.745691 (+ 410us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:46.745711 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:46.747433 (+ 1722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:46.747438 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:46.749296 (+ 1858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:46.749310 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:46.750981 (+ 1671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:46.750988 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:46.751114 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:46.751136 (+ 22us) client_negotiation.cc:769] Sending connection context
0619 16:46:46.751185 (+ 49us) client_negotiation.cc:240] Negotiation successful
0619 16:46:46.751274 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":638,"thread_start_us":488,"threads_started":1}
I20260619 16:46:46.752310 16368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:46.739695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49612 (local address 127.0.97.3:37569)
0619 16:46:46.744752 (+ 5057us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:46.744760 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:46.744790 (+ 30us) server_negotiation.cc:407] Connection header received
0619 16:46:46.744871 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:46.744878 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:46.745062 (+ 184us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:46.745828 (+ 766us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:46:46.746100 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:46.747312 (+ 1212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:46.749452 (+ 2140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:46.750762 (+ 1310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:46.751687 (+ 925us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:46.751898 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0619 16:46:46.751948 (+ 50us) server_negotiation.cc:299] Negotiation successful
0619 16:46:46.752056 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4888,"thread_start_us":4745,"threads_started":1}
I20260619 16:46:46.754444 16023 tablet_service.cc:1467] Processing CreateTablet for tablet 2999f5cbe0e54ffb81f5e4e6a8b75da5 (DEFAULT_TABLE table=test-table [id=6cb6600f212340cca5779fd9d6f3c8df]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:46:46.754446 16157 tablet_service.cc:1467] Processing CreateTablet for tablet 2999f5cbe0e54ffb81f5e4e6a8b75da5 (DEFAULT_TABLE table=test-table [id=6cb6600f212340cca5779fd9d6f3c8df]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:46:46.755861 16157 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2999f5cbe0e54ffb81f5e4e6a8b75da5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:46:46.755867 16023 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2999f5cbe0e54ffb81f5e4e6a8b75da5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:46:46.755491 16290 tablet_service.cc:1467] Processing CreateTablet for tablet 2999f5cbe0e54ffb81f5e4e6a8b75da5 (DEFAULT_TABLE table=test-table [id=6cb6600f212340cca5779fd9d6f3c8df]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:46:46.756601 16290 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2999f5cbe0e54ffb81f5e4e6a8b75da5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:46:46.766957 16373 tablet_bootstrap.cc:492] T 2999f5cbe0e54ffb81f5e4e6a8b75da5 P 1efaebd5eb7a4bba9df40e4b0093546c: Bootstrap starting.
I20260619 16:46:46.769559 16374 tablet_bootstrap.cc:492] T 2999f5cbe0e54ffb81f5e4e6a8b75da5 P 8b323dc8ce0f412db10a79763cb324ce: Bootstrap starting.
I20260619 16:46:46.770306 16375 tablet_bootstrap.cc:492] T 2999f5cbe0e54ffb81f5e4e6a8b75da5 P be15a6b9c4824e229834519ec2b988d0: Bootstrap starting.
I20260619 16:46:46.770932 16373 tablet_bootstrap.cc:654] T 2999f5cbe0e54ffb81f5e4e6a8b75da5 P 1efaebd5eb7a4bba9df40e4b0093546c: Neither blocks nor log segments found. Creating new log.
I20260619 16:46:46.772148 16373 log.cc:824] T 2999f5cbe0e54ffb81f5e4e6a8b75da5 P 1efaebd5eb7a4bba9df40e4b0093546c: Log is configured to *not* fsync() on all Append() calls
I20260619 16:46:46.773599 16375 tablet_bootstrap.cc:654] T 2999f5cbe0e54ffb81f5e4e6a8b75da5 P be15a6b9c4824e229834519ec2b988d0: Neither blocks nor log segments found. Creating new log.
I20260619 16:46:46.774653 16373 tablet_bootstrap.cc:492] T 2999f5cbe0e54ffb81f5e4e6a8b75da5 P 1efaebd5eb7a4bba9df40e4b0093546c: No bootstrap required, opened a new log
I20260619 16:46:46.774796 16375 log.cc:824] T 2999f5cbe0e54ffb81f5e4e6a8b75da5 P be15a6b9c4824e229834519ec2b988d0: Log is configured to *not* fsync() on all Append() calls
I20260619 16:46:46.774892 16373 ts_tablet_manager.cc:1397] T 2999f5cbe0e54ffb81f5e4e6a8b75da5 P 1efaebd5eb7a4bba9df40e4b0093546c: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260619 16:46:46.775817 16374 tablet_bootstrap.cc:654] T 2999f5cbe0e54ffb81f5e4e6a8b75da5 P 8b323dc8ce0f412db10a79763cb324ce: Neither blocks nor log segments found. Creating new log.
I20260619 16:46:46.776928 16374 log.cc:824] T 2999f5cbe0e54ffb81f5e4e6a8b75da5 P 8b323dc8ce0f412db10a79763cb324ce: Log is configured to *not* fsync() on all Append() calls
I20260619 16:46:46.777573 16375 tablet_bootstrap.cc:492] T 2999f5cbe0e54ffb81f5e4e6a8b75da5 P be15a6b9c4824e229834519ec2b988d0: No bootstrap required, opened a new log
I20260619 16:46:46.777853 16375 ts_tablet_manager.cc:1397] T 2999f5cbe0e54ffb81f5e4e6a8b75da5 P be15a6b9c4824e229834519ec2b988d0: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.005s
I20260619 16:46:46.779373 16374 tablet_bootstrap.cc:492] T 2999f5cbe0e54ffb81f5e4e6a8b75da5 P 8b323dc8ce0f412db10a79763cb324ce: No bootstrap required, opened a new log
I20260619 16:46:46.779639 16374 ts_tablet_manager.cc:1397] T 2999f5cbe0e54ffb81f5e4e6a8b75da5 P 8b323dc8ce0f412db10a79763cb324ce: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260619 16:46:46.785040 16373 raft_consensus.cc:348] T 2999f5cbe0e54ffb81f5e4e6a8b75da5 P 1efaebd5eb7a4bba9df40e4b0093546c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1efaebd5eb7a4bba9df40e4b0093546c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37569 } } peers { permanent_uuid: "8b323dc8ce0f412db10a79763cb324ce" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34415 } } peers { permanent_uuid: "be15a6b9c4824e229834519ec2b988d0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42315 } }
I20260619 16:46:46.785461 16373 raft_consensus.cc:374] T 2999f5cbe0e54ffb81f5e4e6a8b75da5 P 1efaebd5eb7a4bba9df40e4b0093546c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:46:46.785590 16373 raft_consensus.cc:729] T 2999f5cbe0e54ffb81f5e4e6a8b75da5 P 1efaebd5eb7a4bba9df40e4b0093546c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1efaebd5eb7a4bba9df40e4b0093546c, State: Initialized, Role: FOLLOWER
I20260619 16:46:46.786126 16374 raft_consensus.cc:348] T 2999f5cbe0e54ffb81f5e4e6a8b75da5 P 8b323dc8ce0f412db10a79763cb324ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1efaebd5eb7a4bba9df40e4b0093546c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37569 } } peers { permanent_uuid: "8b323dc8ce0f412db10a79763cb324ce" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34415 } } peers { permanent_uuid: "be15a6b9c4824e229834519ec2b988d0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42315 } }
I20260619 16:46:46.786223 16373 consensus_queue.cc:260] T 2999f5cbe0e54ffb81f5e4e6a8b75da5 P 1efaebd5eb7a4bba9df40e4b0093546c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1efaebd5eb7a4bba9df40e4b0093546c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37569 } } peers { permanent_uuid: "8b323dc8ce0f412db10a79763cb324ce" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34415 } } peers { permanent_uuid: "be15a6b9c4824e229834519ec2b988d0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42315 } }
I20260619 16:46:46.786510 16374 raft_consensus.cc:374] T 2999f5cbe0e54ffb81f5e4e6a8b75da5 P 8b323dc8ce0f412db10a79763cb324ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:46:46.786612 16374 raft_consensus.cc:729] T 2999f5cbe0e54ffb81f5e4e6a8b75da5 P 8b323dc8ce0f412db10a79763cb324ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b323dc8ce0f412db10a79763cb324ce, State: Initialized, Role: FOLLOWER
I20260619 16:46:46.787156 16374 consensus_queue.cc:260] T 2999f5cbe0e54ffb81f5e4e6a8b75da5 P 8b323dc8ce0f412db10a79763cb324ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1efaebd5eb7a4bba9df40e4b0093546c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37569 } } peers { permanent_uuid: "8b323dc8ce0f412db10a79763cb324ce" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34415 } } peers { permanent_uuid: "be15a6b9c4824e229834519ec2b988d0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42315 } }
I20260619 16:46:46.787761 16375 raft_consensus.cc:348] T 2999f5cbe0e54ffb81f5e4e6a8b75da5 P be15a6b9c4824e229834519ec2b988d0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1efaebd5eb7a4bba9df40e4b0093546c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37569 } } peers { permanent_uuid: "8b323dc8ce0f412db10a79763cb324ce" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34415 } } peers { permanent_uuid: "be15a6b9c4824e229834519ec2b988d0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42315 } }
I20260619 16:46:46.788173 16375 raft_consensus.cc:374] T 2999f5cbe0e54ffb81f5e4e6a8b75da5 P be15a6b9c4824e229834519ec2b988d0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:46:46.788264 16375 raft_consensus.cc:729] T 2999f5cbe0e54ffb81f5e4e6a8b75da5 P be15a6b9c4824e229834519ec2b988d0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: be15a6b9c4824e229834519ec2b988d0, State: Initialized, Role: FOLLOWER
I20260619 16:46:46.788839 16375 consensus_queue.cc:260] T 2999f5cbe0e54ffb81f5e4e6a8b75da5 P be15a6b9c4824e229834519ec2b988d0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1efaebd5eb7a4bba9df40e4b0093546c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37569 } } peers { permanent_uuid: "8b323dc8ce0f412db10a79763cb324ce" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34415 } } peers { permanent_uuid: "be15a6b9c4824e229834519ec2b988d0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42315 } }
I20260619 16:46:46.791029 16222 heartbeater.cc:503] Master 127.0.97.62:44977 was elected leader, sending a full tablet report...
I20260619 16:46:46.791934 16374 ts_tablet_manager.cc:1428] T 2999f5cbe0e54ffb81f5e4e6a8b75da5 P 8b323dc8ce0f412db10a79763cb324ce: Time spent starting tablet: real 0.012s user 0.007s sys 0.004s
I20260619 16:46:46.792732 16355 heartbeater.cc:503] Master 127.0.97.62:44977 was elected leader, sending a full tablet report...
I20260619 16:46:46.793497 16373 ts_tablet_manager.cc:1428] T 2999f5cbe0e54ffb81f5e4e6a8b75da5 P 1efaebd5eb7a4bba9df40e4b0093546c: Time spent starting tablet: real 0.018s user 0.015s sys 0.004s
I20260619 16:46:46.794914 16375 ts_tablet_manager.cc:1428] T 2999f5cbe0e54ffb81f5e4e6a8b75da5 P be15a6b9c4824e229834519ec2b988d0: Time spent starting tablet: real 0.017s user 0.009s sys 0.007s
I20260619 16:46:46.846473 16381 raft_consensus.cc:482] T 2999f5cbe0e54ffb81f5e4e6a8b75da5 P be15a6b9c4824e229834519ec2b988d0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 16:46:46.846787 16381 raft_consensus.cc:504] T 2999f5cbe0e54ffb81f5e4e6a8b75da5 P be15a6b9c4824e229834519ec2b988d0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1efaebd5eb7a4bba9df40e4b0093546c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37569 } } peers { permanent_uuid: "8b323dc8ce0f412db10a79763cb324ce" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34415 } } peers { permanent_uuid: "be15a6b9c4824e229834519ec2b988d0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42315 } }
I20260619 16:46:46.848465 16381 leader_election.cc:288] T 2999f5cbe0e54ffb81f5e4e6a8b75da5 P be15a6b9c4824e229834519ec2b988d0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1efaebd5eb7a4bba9df40e4b0093546c (127.0.97.3:37569), 8b323dc8ce0f412db10a79763cb324ce (127.0.97.2:34415)
I20260619 16:46:46.855607 16368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:46.848947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:42013 (local address 127.0.97.3:37569)
0619 16:46:46.849210 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:46.849214 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:46.849673 (+ 459us) server_negotiation.cc:407] Connection header received
0619 16:46:46.849935 (+ 262us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:46.849940 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:46.850017 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:46.850123 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:46:46.850784 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:46.851759 (+ 975us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:46.853728 (+ 1969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:46.854674 (+ 946us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:46.854770 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:46.854854 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0619 16:46:46.855227 (+ 373us) server_negotiation.cc:299] Negotiation successful
0619 16:46:46.855358 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120}
I20260619 16:46:46.855727 16372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:46.849969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:33235 (local address 127.0.97.2:34415)
0619 16:46:46.850219 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:46.850225 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:46.850454 (+ 229us) server_negotiation.cc:407] Connection header received
0619 16:46:46.850579 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:46.850584 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:46.850642 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:46.850733 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:46:46.851198 (+ 465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:46.852081 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:46.853946 (+ 1865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:46.854865 (+ 919us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:46.854998 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:46.855085 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0619 16:46:46.855357 (+ 272us) server_negotiation.cc:299] Negotiation successful
0619 16:46:46.855486 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260619 16:46:46.855762 16382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:46.848882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:37569 (local address 127.0.97.1:42013)
0619 16:46:46.849541 (+ 659us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:46.849560 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 16:46:46.849703 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:46:46.850236 (+ 533us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:46:46.850241 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:46:46.850268 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:46:46.850642 (+ 374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:46.850652 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:46.851901 (+ 1249us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:46.851905 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:46.853549 (+ 1644us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:46.853578 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:46.854859 (+ 1281us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:46.854866 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:46.855033 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:46.855068 (+ 35us) client_negotiation.cc:769] Sending connection context
0619 16:46:46.855152 (+ 84us) client_negotiation.cc:240] Negotiation successful
0619 16:46:46.855280 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":533,"mutex_wait_us":15,"thread_start_us":354,"threads_started":1}
I20260619 16:46:46.855896 16383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:46.849719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:34415 (local address 127.0.97.1:33235)
0619 16:46:46.850333 (+ 614us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:46.850349 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 16:46:46.850473 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:46:46.850755 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:46:46.850758 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:46:46.850786 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:46:46.851040 (+ 254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:46.851047 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:46.852223 (+ 1176us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:46.852229 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:46.853801 (+ 1572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:46.853816 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:46.855057 (+ 1241us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:46.855062 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:46.855202 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:46.855234 (+ 32us) client_negotiation.cc:769] Sending connection context
0619 16:46:46.855322 (+ 88us) client_negotiation.cc:240] Negotiation successful
0619 16:46:46.855437 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":502,"mutex_wait_us":22,"thread_start_us":263,"threads_started":1}
I20260619 16:46:46.856902 16177 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2999f5cbe0e54ffb81f5e4e6a8b75da5" candidate_uuid: "be15a6b9c4824e229834519ec2b988d0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b323dc8ce0f412db10a79763cb324ce" is_pre_election: true
I20260619 16:46:46.856901 16310 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2999f5cbe0e54ffb81f5e4e6a8b75da5" candidate_uuid: "be15a6b9c4824e229834519ec2b988d0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1efaebd5eb7a4bba9df40e4b0093546c" is_pre_election: true
I20260619 16:46:46.857372 16310 raft_consensus.cc:2454] T 2999f5cbe0e54ffb81f5e4e6a8b75da5 P 1efaebd5eb7a4bba9df40e4b0093546c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate be15a6b9c4824e229834519ec2b988d0 in term 0.
I20260619 16:46:46.857372 16177 raft_consensus.cc:2454] T 2999f5cbe0e54ffb81f5e4e6a8b75da5 P 8b323dc8ce0f412db10a79763cb324ce [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate be15a6b9c4824e229834519ec2b988d0 in term 0.
I20260619 16:46:46.857990 15978 leader_election.cc:302] T 2999f5cbe0e54ffb81f5e4e6a8b75da5 P be15a6b9c4824e229834519ec2b988d0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1efaebd5eb7a4bba9df40e4b0093546c, be15a6b9c4824e229834519ec2b988d0; no voters:
I20260619 16:46:46.858359 16381 raft_consensus.cc:2781] T 2999f5cbe0e54ffb81f5e4e6a8b75da5 P be15a6b9c4824e229834519ec2b988d0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 16:46:46.858495 16381 raft_consensus.cc:482] T 2999f5cbe0e54ffb81f5e4e6a8b75da5 P be15a6b9c4824e229834519ec2b988d0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 16:46:46.858584 16381 raft_consensus.cc:3037] T 2999f5cbe0e54ffb81f5e4e6a8b75da5 P be15a6b9c4824e229834519ec2b988d0 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:46:46.860931 16381 raft_consensus.cc:504] T 2999f5cbe0e54ffb81f5e4e6a8b75da5 P be15a6b9c4824e229834519ec2b988d0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1efaebd5eb7a4bba9df40e4b0093546c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37569 } } peers { permanent_uuid: "8b323dc8ce0f412db10a79763cb324ce" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34415 } } peers { permanent_uuid: "be15a6b9c4824e229834519ec2b988d0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42315 } }
I20260619 16:46:46.861618 16381 leader_election.cc:288] T 2999f5cbe0e54ffb81f5e4e6a8b75da5 P be15a6b9c4824e229834519ec2b988d0 [CANDIDATE]: Term 1 election: Requested vote from peers 1efaebd5eb7a4bba9df40e4b0093546c (127.0.97.3:37569), 8b323dc8ce0f412db10a79763cb324ce (127.0.97.2:34415)
I20260619 16:46:46.862051 16310 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2999f5cbe0e54ffb81f5e4e6a8b75da5" candidate_uuid: "be15a6b9c4824e229834519ec2b988d0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1efaebd5eb7a4bba9df40e4b0093546c"
I20260619 16:46:46.862155 16177 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2999f5cbe0e54ffb81f5e4e6a8b75da5" candidate_uuid: "be15a6b9c4824e229834519ec2b988d0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b323dc8ce0f412db10a79763cb324ce"
I20260619 16:46:46.862272 16310 raft_consensus.cc:3037] T 2999f5cbe0e54ffb81f5e4e6a8b75da5 P 1efaebd5eb7a4bba9df40e4b0093546c [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:46:46.862385 16177 raft_consensus.cc:3037] T 2999f5cbe0e54ffb81f5e4e6a8b75da5 P 8b323dc8ce0f412db10a79763cb324ce [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:46:46.864434 16177 raft_consensus.cc:2454] T 2999f5cbe0e54ffb81f5e4e6a8b75da5 P 8b323dc8ce0f412db10a79763cb324ce [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate be15a6b9c4824e229834519ec2b988d0 in term 1.
I20260619 16:46:46.864440 16310 raft_consensus.cc:2454] T 2999f5cbe0e54ffb81f5e4e6a8b75da5 P 1efaebd5eb7a4bba9df40e4b0093546c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate be15a6b9c4824e229834519ec2b988d0 in term 1.
I20260619 16:46:46.864800 15978 leader_election.cc:302] T 2999f5cbe0e54ffb81f5e4e6a8b75da5 P be15a6b9c4824e229834519ec2b988d0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1efaebd5eb7a4bba9df40e4b0093546c, be15a6b9c4824e229834519ec2b988d0; no voters:
I20260619 16:46:46.865051 16381 raft_consensus.cc:2781] T 2999f5cbe0e54ffb81f5e4e6a8b75da5 P be15a6b9c4824e229834519ec2b988d0 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:46:46.865453 16381 raft_consensus.cc:686] T 2999f5cbe0e54ffb81f5e4e6a8b75da5 P be15a6b9c4824e229834519ec2b988d0 [term 1 LEADER]: Becoming Leader. State: Replica: be15a6b9c4824e229834519ec2b988d0, State: Running, Role: LEADER
I20260619 16:46:46.865936 16381 consensus_queue.cc:237] T 2999f5cbe0e54ffb81f5e4e6a8b75da5 P be15a6b9c4824e229834519ec2b988d0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1efaebd5eb7a4bba9df40e4b0093546c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37569 } } peers { permanent_uuid: "8b323dc8ce0f412db10a79763cb324ce" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34415 } } peers { permanent_uuid: "be15a6b9c4824e229834519ec2b988d0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42315 } }
I20260619 16:46:46.870461 15897 catalog_manager.cc:5495] T 2999f5cbe0e54ffb81f5e4e6a8b75da5 P be15a6b9c4824e229834519ec2b988d0 reported cstate change: term changed from 0 to 1, leader changed from <none> to be15a6b9c4824e229834519ec2b988d0 (127.0.97.1). New cstate: current_term: 1 leader_uuid: "be15a6b9c4824e229834519ec2b988d0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1efaebd5eb7a4bba9df40e4b0093546c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37569 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8b323dc8ce0f412db10a79763cb324ce" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34415 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "be15a6b9c4824e229834519ec2b988d0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42315 } health_report { overall_health: HEALTHY } } }
I20260619 16:46:46.903766 16370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:46.898531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40582 (local address 127.0.97.1:42315)
0619 16:46:46.898729 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:46.898733 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:46.898777 (+ 44us) server_negotiation.cc:407] Connection header received
0619 16:46:46.898989 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:46.898993 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:46.899100 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:46.899242 (+ 142us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 16:46:46.899522 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:46.900237 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:46.901599 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:46.902102 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:46.902213 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:46.902706 (+ 493us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 16:46:46.903252 (+ 546us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 16:46:46.903366 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0619 16:46:46.903442 (+ 76us) server_negotiation.cc:299] Negotiation successful
0619 16:46:46.903510 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260619 16:46:46.911636 16310 raft_consensus.cc:1261] T 2999f5cbe0e54ffb81f5e4e6a8b75da5 P 1efaebd5eb7a4bba9df40e4b0093546c [term 1 FOLLOWER]: Refusing update from remote peer be15a6b9c4824e229834519ec2b988d0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:46:46.911653 16177 raft_consensus.cc:1261] T 2999f5cbe0e54ffb81f5e4e6a8b75da5 P 8b323dc8ce0f412db10a79763cb324ce [term 1 FOLLOWER]: Refusing update from remote peer be15a6b9c4824e229834519ec2b988d0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:46:46.912451 16384 consensus_queue.cc:1035] T 2999f5cbe0e54ffb81f5e4e6a8b75da5 P be15a6b9c4824e229834519ec2b988d0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1efaebd5eb7a4bba9df40e4b0093546c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37569 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:46:46.912768 16381 consensus_queue.cc:1035] T 2999f5cbe0e54ffb81f5e4e6a8b75da5 P be15a6b9c4824e229834519ec2b988d0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b323dc8ce0f412db10a79763cb324ce" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34415 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260619 16:46:46.920899 16356 tablet.cc:2369] T 2999f5cbe0e54ffb81f5e4e6a8b75da5 P 1efaebd5eb7a4bba9df40e4b0093546c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 16:46:46.924208 16389 mvcc.cc:205] Tried to move back new op lower bound from 7298611637900664832 to 7298611637726330880. Current Snapshot: MvccSnapshot[applied={T|T < 7298611637900664832}]
I20260619 16:46:46.935215 15899 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40972:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:46:46.935447 15899 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40972:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:46:46.938668 15899 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 460891f6ca034e7c99713cb56b72d30f: Sending DeleteTablet for 3 replicas of tablet 2999f5cbe0e54ffb81f5e4e6a8b75da5
I20260619 16:46:46.939644 16290 tablet_service.cc:1514] Processing DeleteTablet for tablet 2999f5cbe0e54ffb81f5e4e6a8b75da5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:46:46 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:49612
I20260619 16:46:46.939752 16157 tablet_service.cc:1514] Processing DeleteTablet for tablet 2999f5cbe0e54ffb81f5e4e6a8b75da5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:46:46 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:40428
I20260619 16:46:46.939803 16023 tablet_service.cc:1514] Processing DeleteTablet for tablet 2999f5cbe0e54ffb81f5e4e6a8b75da5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:46:46 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:40578
I20260619 16:46:46.940985 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu pid 15960
I20260619 16:46:46.951004 16399 tablet_replica.cc:330] T 2999f5cbe0e54ffb81f5e4e6a8b75da5 P be15a6b9c4824e229834519ec2b988d0: stopping tablet replica
I20260619 16:46:46.951534 16399 raft_consensus.cc:2229] T 2999f5cbe0e54ffb81f5e4e6a8b75da5 P be15a6b9c4824e229834519ec2b988d0 [term 1 LEADER]: Raft consensus shutting down.
I20260619 16:46:46.952376 16399 raft_consensus.cc:2258] T 2999f5cbe0e54ffb81f5e4e6a8b75da5 P be15a6b9c4824e229834519ec2b988d0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:46:46.954771 16399 ts_tablet_manager.cc:1905] T 2999f5cbe0e54ffb81f5e4e6a8b75da5 P be15a6b9c4824e229834519ec2b988d0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:46:46.958338 16398 tablet_replica.cc:330] T 2999f5cbe0e54ffb81f5e4e6a8b75da5 P 8b323dc8ce0f412db10a79763cb324ce: stopping tablet replica
I20260619 16:46:46.958799 16398 raft_consensus.cc:2229] T 2999f5cbe0e54ffb81f5e4e6a8b75da5 P 8b323dc8ce0f412db10a79763cb324ce [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 16:46:46.959362 16397 tablet_replica.cc:330] T 2999f5cbe0e54ffb81f5e4e6a8b75da5 P 1efaebd5eb7a4bba9df40e4b0093546c: stopping tablet replica
I20260619 16:46:46.959401 16398 raft_consensus.cc:2258] T 2999f5cbe0e54ffb81f5e4e6a8b75da5 P 8b323dc8ce0f412db10a79763cb324ce [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:46:46.959762 16397 raft_consensus.cc:2229] T 2999f5cbe0e54ffb81f5e4e6a8b75da5 P 1efaebd5eb7a4bba9df40e4b0093546c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 16:46:46.960220 16397 raft_consensus.cc:2258] T 2999f5cbe0e54ffb81f5e4e6a8b75da5 P 1efaebd5eb7a4bba9df40e4b0093546c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:46:46.961865 16398 ts_tablet_manager.cc:1905] T 2999f5cbe0e54ffb81f5e4e6a8b75da5 P 8b323dc8ce0f412db10a79763cb324ce: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:46:46.962338 16397 ts_tablet_manager.cc:1905] T 2999f5cbe0e54ffb81f5e4e6a8b75da5 P 1efaebd5eb7a4bba9df40e4b0093546c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:46:46.965332 16399 ts_tablet_manager.cc:1918] T 2999f5cbe0e54ffb81f5e4e6a8b75da5 P be15a6b9c4824e229834519ec2b988d0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 16:46:46.965503 16399 log.cc:1196] T 2999f5cbe0e54ffb81f5e4e6a8b75da5 P be15a6b9c4824e229834519ec2b988d0: Deleting WAL directory at /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887385903558-388-0/minicluster-data/ts-0/wal/wals/2999f5cbe0e54ffb81f5e4e6a8b75da5
I20260619 16:46:46.966048 16399 ts_tablet_manager.cc:1939] T 2999f5cbe0e54ffb81f5e4e6a8b75da5 P be15a6b9c4824e229834519ec2b988d0: Deleting consensus metadata
I20260619 16:46:46.967940 15883 catalog_manager.cc:4841] TS be15a6b9c4824e229834519ec2b988d0 (127.0.97.1:42315): tablet 2999f5cbe0e54ffb81f5e4e6a8b75da5 (table test-table [id=6cb6600f212340cca5779fd9d6f3c8df]) successfully deleted
I20260619 16:46:46.968611 16397 ts_tablet_manager.cc:1918] T 2999f5cbe0e54ffb81f5e4e6a8b75da5 P 1efaebd5eb7a4bba9df40e4b0093546c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 16:46:46.968753 16397 log.cc:1196] T 2999f5cbe0e54ffb81f5e4e6a8b75da5 P 1efaebd5eb7a4bba9df40e4b0093546c: Deleting WAL directory at /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887385903558-388-0/minicluster-data/ts-2/wal/wals/2999f5cbe0e54ffb81f5e4e6a8b75da5
I20260619 16:46:46.969235 16397 ts_tablet_manager.cc:1939] T 2999f5cbe0e54ffb81f5e4e6a8b75da5 P 1efaebd5eb7a4bba9df40e4b0093546c: Deleting consensus metadata
I20260619 16:46:46.969497 16398 ts_tablet_manager.cc:1918] T 2999f5cbe0e54ffb81f5e4e6a8b75da5 P 8b323dc8ce0f412db10a79763cb324ce: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 16:46:46.969707 16398 log.cc:1196] T 2999f5cbe0e54ffb81f5e4e6a8b75da5 P 8b323dc8ce0f412db10a79763cb324ce: Deleting WAL directory at /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887385903558-388-0/minicluster-data/ts-1/wal/wals/2999f5cbe0e54ffb81f5e4e6a8b75da5
I20260619 16:46:46.970228 16398 ts_tablet_manager.cc:1939] T 2999f5cbe0e54ffb81f5e4e6a8b75da5 P 8b323dc8ce0f412db10a79763cb324ce: Deleting consensus metadata
I20260619 16:46:46.971352 15885 catalog_manager.cc:4841] TS 1efaebd5eb7a4bba9df40e4b0093546c (127.0.97.3:37569): tablet 2999f5cbe0e54ffb81f5e4e6a8b75da5 (table test-table [id=6cb6600f212340cca5779fd9d6f3c8df]) successfully deleted
I20260619 16:46:46.971812 15885 catalog_manager.cc:4841] TS 8b323dc8ce0f412db10a79763cb324ce (127.0.97.2:34415): tablet 2999f5cbe0e54ffb81f5e4e6a8b75da5 (table test-table [id=6cb6600f212340cca5779fd9d6f3c8df]) successfully deleted
Jun 19 16:46:46 dist-test-slave-wvz7 krb5kdc[15854](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887604, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260619 16:46:46.980156 16370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:46.942108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40598 (local address 127.0.97.1:42315)
0619 16:46:46.942300 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:46.942307 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:46.958720 (+ 16413us) server_negotiation.cc:407] Connection header received
0619 16:46:46.958928 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:46.958933 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:46.959014 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:46.959117 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:46:46.961120 (+ 2003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:46.962477 (+ 1357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:46.967825 (+ 5348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:46.968519 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:46.968619 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:46.974334 (+ 5715us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:46:46.974347 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:46:46.974361 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:46:46.974397 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:46:46.977238 (+ 2841us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:46.978120 (+ 882us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:46.978131 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:46.978134 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:46.978304 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:46.978756 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:46.978759 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:46.978761 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:46.979088 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:46:46.979304 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0619 16:46:46.979682 (+ 378us) server_negotiation.cc:299] Negotiation successful
0619 16:46:46.979859 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260619 16:46:46.980803 16083 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:46:47.079380 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu with pid 15960
I20260619 16:46:47.094067 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu pid 16094
Jun 19 16:46:47 dist-test-slave-wvz7 krb5kdc[15854](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887604, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260619 16:46:47.108321 16372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:47.094811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40440 (local address 127.0.97.2:34415)
0619 16:46:47.094974 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:47.094979 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:47.094997 (+ 18us) server_negotiation.cc:407] Connection header received
0619 16:46:47.095148 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:47.095152 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:47.095243 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:47.095325 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:46:47.096506 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:47.097218 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:47.098483 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:47.098864 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:47.098951 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:47.103204 (+ 4253us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:46:47.103218 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:46:47.103234 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:46:47.103276 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:46:47.106032 (+ 2756us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:47.106612 (+ 580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:47.106618 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:47.106620 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:47.106732 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:47.107012 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:47.107015 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:47.107017 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:47.107406 (+ 389us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:46:47.107625 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0619 16:46:47.107845 (+ 220us) server_negotiation.cc:299] Negotiation successful
0619 16:46:47.107993 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260619 16:46:47.108954 16217 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:46:47.196560 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu with pid 16094
I20260619 16:46:47.210695 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu pid 16227
Jun 19 16:46:47 dist-test-slave-wvz7 krb5kdc[15854](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887604, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260619 16:46:47.226279 16368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:47.211432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49628 (local address 127.0.97.3:37569)
0619 16:46:47.211626 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:47.211633 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:47.211656 (+ 23us) server_negotiation.cc:407] Connection header received
0619 16:46:47.211875 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:47.211880 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:47.211951 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:47.212057 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:46:47.213456 (+ 1399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:47.214285 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:47.215792 (+ 1507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:47.216218 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:47.216346 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:47.221070 (+ 4724us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:46:47.221089 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:46:47.221111 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:46:47.221151 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:46:47.223886 (+ 2735us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:47.224601 (+ 715us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:47.224609 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:47.224611 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:47.224752 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:47.225038 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:47.225041 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:47.225043 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:47.225385 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:46:47.225611 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0619 16:46:47.225865 (+ 254us) server_negotiation.cc:299] Negotiation successful
0619 16:46:47.226031 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260619 16:46:47.226902 16350 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:46:47.315160 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu with pid 16227
I20260619 16:46:47.330068 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu pid 15867
I20260619 16:46:47.331238 15929 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:46:47.439448 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu with pid 15867
I20260619 16:47:07.456537 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu
/tmp/dist-test-taskEs2EH8/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887385903558-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887385903558-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887385903558-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887385903558-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:44977
--webserver_interface=127.0.97.62
--webserver_port=43343
--keytab_file=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887385903558-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45131
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:44977
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887385903558-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887385903558-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887385903558-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:47:07.768664 16405 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:47:07.768934 16405 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:47:07.769048 16405 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:47:07.776245 16405 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260619 16:47:07.776357 16405 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260619 16:47:07.776401 16405 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260619 16:47:07.776469 16405 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:47:07.776535 16405 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:47:07.776638 16405 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:47:07.776692 16405 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:47:07.776764 16405 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:47:07.786597 16405 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45131
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887385903558-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887385903558-388-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:44977
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887385903558-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:44977
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887385903558-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=43343
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887385903558-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:47:07.787812 16405 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:47:07.789086 16405 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:47:07.809618 16414 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:47:07.809644 16412 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:47:07.809623 16411 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:47:07.810302 16405 server_base.cc:1034] running on GCE node
I20260619 16:47:07.811208 16405 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:47:07.812770 16405 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:47:07.813970 16405 hybrid_clock.cc:648] HybridClock initialized: now 1781887627813893 us; error 48 us; skew 500 ppm
I20260619 16:47:07.814342 16405 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:47:07 dist-test-slave-wvz7 krb5kdc[15854](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887627, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:47:07.817875 16405 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260619 16:47:07.819339 16405 webserver.cc:460] Webserver started at http://127.0.97.62:43343/ using document root <none> and password file <none>
I20260619 16:47:07.820070 16405 fs_manager.cc:359] Metadata directory not provided
I20260619 16:47:07.820175 16405 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:47:07.826038 16405 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260619 16:47:07.828892 16421 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:47:07.830256 16405 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.003s
I20260619 16:47:07.830415 16405 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887385903558-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887385903558-388-0/minicluster-data/master-0/wal
uuid: "460891f6ca034e7c99713cb56b72d30f"
format_stamp: "Formatted at 2026-06-19 16:46:45 on dist-test-slave-wvz7"
server_key: "45c073fb68fba1d7d41be8c7cb093973"
server_key_iv: "5428244b0fcbb52060183807b9da9d45"
server_key_version: "encryptionkey@0"
I20260619 16:47:07.831410 16405 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887385903558-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887385903558-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887385903558-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:47:07.844246 16405 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:47:07.848542 16405 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:47:07.848812 16405 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:47:07.865981 16405 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:44977
I20260619 16:47:07.865999 16472 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:44977 every 8 connection(s)
I20260619 16:47:07.867527 16405 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887385903558-388-0/minicluster-data/master-0/data/info.pb
I20260619 16:47:07.874225 16473 sys_catalog.cc:263] Verifying existing consensus state
I20260619 16:47:07.877426 16473 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 460891f6ca034e7c99713cb56b72d30f: Bootstrap starting.
I20260619 16:47:07.877751 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu as pid 16405
I20260619 16:47:07.878516 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu
/tmp/dist-test-taskEs2EH8/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887385903558-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887385903558-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887385903558-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887385903558-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:42315
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:44977
--webserver_port=43223
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887385903558-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45131
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887385903558-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887385903558-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887385903558-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 16:47:07.899060 16473 log.cc:824] T 00000000000000000000000000000000 P 460891f6ca034e7c99713cb56b72d30f: Log is configured to *not* fsync() on all Append() calls
I20260619 16:47:07.910688 16473 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 460891f6ca034e7c99713cb56b72d30f: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 16:47:07.911185 16473 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 460891f6ca034e7c99713cb56b72d30f: Bootstrap complete.
I20260619 16:47:07.919883 16473 raft_consensus.cc:348] T 00000000000000000000000000000000 P 460891f6ca034e7c99713cb56b72d30f [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "460891f6ca034e7c99713cb56b72d30f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44977 } }
I20260619 16:47:07.920706 16473 raft_consensus.cc:729] T 00000000000000000000000000000000 P 460891f6ca034e7c99713cb56b72d30f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 460891f6ca034e7c99713cb56b72d30f, State: Initialized, Role: FOLLOWER
I20260619 16:47:07.921304 16473 consensus_queue.cc:260] T 00000000000000000000000000000000 P 460891f6ca034e7c99713cb56b72d30f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "460891f6ca034e7c99713cb56b72d30f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44977 } }
I20260619 16:47:07.921523 16473 raft_consensus.cc:388] T 00000000000000000000000000000000 P 460891f6ca034e7c99713cb56b72d30f [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:47:07.921625 16473 raft_consensus.cc:482] T 00000000000000000000000000000000 P 460891f6ca034e7c99713cb56b72d30f [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:47:07.921805 16473 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 460891f6ca034e7c99713cb56b72d30f [term 1 FOLLOWER]: Advancing to term 2
I20260619 16:47:07.925387 16473 raft_consensus.cc:504] T 00000000000000000000000000000000 P 460891f6ca034e7c99713cb56b72d30f [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "460891f6ca034e7c99713cb56b72d30f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44977 } }
I20260619 16:47:07.925900 16473 leader_election.cc:302] T 00000000000000000000000000000000 P 460891f6ca034e7c99713cb56b72d30f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 460891f6ca034e7c99713cb56b72d30f; no voters:
I20260619 16:47:07.926507 16473 leader_election.cc:288] T 00000000000000000000000000000000 P 460891f6ca034e7c99713cb56b72d30f [CANDIDATE]: Term 2 election: Requested vote from peers
I20260619 16:47:07.926632 16477 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 460891f6ca034e7c99713cb56b72d30f [term 2 FOLLOWER]: Leader election won for term 2
I20260619 16:47:07.927753 16477 raft_consensus.cc:686] T 00000000000000000000000000000000 P 460891f6ca034e7c99713cb56b72d30f [term 2 LEADER]: Becoming Leader. State: Replica: 460891f6ca034e7c99713cb56b72d30f, State: Running, Role: LEADER
I20260619 16:47:07.928349 16477 consensus_queue.cc:237] T 00000000000000000000000000000000 P 460891f6ca034e7c99713cb56b72d30f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "460891f6ca034e7c99713cb56b72d30f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44977 } }
I20260619 16:47:07.928584 16473 sys_catalog.cc:564] T 00000000000000000000000000000000 P 460891f6ca034e7c99713cb56b72d30f [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:47:07.931103 16478 sys_catalog.cc:455] T 00000000000000000000000000000000 P 460891f6ca034e7c99713cb56b72d30f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "460891f6ca034e7c99713cb56b72d30f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "460891f6ca034e7c99713cb56b72d30f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44977 } } }
I20260619 16:47:07.931463 16478 sys_catalog.cc:458] T 00000000000000000000000000000000 P 460891f6ca034e7c99713cb56b72d30f [sys.catalog]: This master's current role is: LEADER
I20260619 16:47:07.936587 16479 sys_catalog.cc:455] T 00000000000000000000000000000000 P 460891f6ca034e7c99713cb56b72d30f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 460891f6ca034e7c99713cb56b72d30f. Latest consensus state: current_term: 2 leader_uuid: "460891f6ca034e7c99713cb56b72d30f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "460891f6ca034e7c99713cb56b72d30f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44977 } } }
I20260619 16:47:07.937407 16479 sys_catalog.cc:458] T 00000000000000000000000000000000 P 460891f6ca034e7c99713cb56b72d30f [sys.catalog]: This master's current role is: LEADER
I20260619 16:47:07.939000 16483 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:47:07.947093 16483 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:47:07.948498 16483 catalog_manager.cc:1259] Loaded cluster ID: 5075d572c7964b97b3497f85f2f1032f
I20260619 16:47:07.948680 16483 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:47:07.952790 16483 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:47:07.954334 16483 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 460891f6ca034e7c99713cb56b72d30f: Loaded TSK: 0
I20260619 16:47:07.966879 16494 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260619 16:47:07.984268 16483 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 460891f6ca034e7c99713cb56b72d30f: Generated new TSK 1
I20260619 16:47:07.984676 16483 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260619 16:47:08.212436 16475 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:47:08.212705 16475 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:47:08.212832 16475 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:47:08.219928 16475 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:47:08.220039 16475 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:47:08.220192 16475 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260619 16:47:08.229761 16475 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45131
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887385903558-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887385903558-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887385903558-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:42315
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887385903558-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=43223
--tserver_master_addrs=127.0.97.62:44977
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887385903558-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:47:08.230964 16475 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:47:08.232208 16475 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:47:08.246415 16501 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:47:08.246842 16500 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:47:08.247205 16503 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:47:08.247479 16475 server_base.cc:1034] running on GCE node
I20260619 16:47:08.248034 16475 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:47:08.248931 16475 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:47:08.250133 16475 hybrid_clock.cc:648] HybridClock initialized: now 1781887628250058 us; error 57 us; skew 500 ppm
I20260619 16:47:08.250497 16475 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:47:08 dist-test-slave-wvz7 krb5kdc[15854](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887628, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:47:08.253489 16475 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260619 16:47:08.254982 16475 webserver.cc:460] Webserver started at http://127.0.97.1:43223/ using document root <none> and password file <none>
I20260619 16:47:08.255657 16475 fs_manager.cc:359] Metadata directory not provided
I20260619 16:47:08.255785 16475 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:47:08.260883 16475 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 16:47:08.263572 16510 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:47:08.264989 16475 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 16:47:08.265126 16475 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887385903558-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887385903558-388-0/minicluster-data/ts-0/wal
uuid: "be15a6b9c4824e229834519ec2b988d0"
format_stamp: "Formatted at 2026-06-19 16:46:45 on dist-test-slave-wvz7"
server_key: "3c2545cdbeb192463e0a296228190a40"
server_key_iv: "8d196395b19860edfa33881c321b1719"
server_key_version: "encryptionkey@0"
I20260619 16:47:08.266116 16475 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887385903558-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887385903558-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887385903558-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:47:08.280875 16475 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:47:08.284860 16475 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:47:08.285108 16475 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:47:08.286087 16475 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:47:08.287370 16475 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:47:08.287460 16475 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:47:08.287563 16475 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:47:08.287631 16475 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:47:08.316896 16475 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:42315
I20260619 16:47:08.316948 16622 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:42315 every 8 connection(s)
I20260619 16:47:08.318312 16475 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887385903558-388-0/minicluster-data/ts-0/data/info.pb
I20260619 16:47:08.324436 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu as pid 16475
I20260619 16:47:08.325155 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu
/tmp/dist-test-taskEs2EH8/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887385903558-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887385903558-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887385903558-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887385903558-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:34415
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:44977
--webserver_port=36797
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887385903558-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45131
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887385903558-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887385903558-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887385903558-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 16:47:08 dist-test-slave-wvz7 krb5kdc[15854](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887628, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 16:47:08.341872 16626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:47:08.321037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44977 (local address 127.0.97.1:43881)
0619 16:47:08.321807 (+ 770us) negotiation.cc:107] Waiting for socket to connect
0619 16:47:08.321843 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0619 16:47:08.323414 (+ 1571us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:47:08.324764 (+ 1350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:47:08.324774 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:47:08.325469 (+ 695us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:47:08.326208 (+ 739us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:47:08.326233 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:47:08.327867 (+ 1634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:47:08.327871 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:47:08.329639 (+ 1768us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:47:08.329654 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:47:08.330728 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:47:08.330771 (+ 43us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:47:08.331031 (+ 260us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:47:08.331654 (+ 623us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:47:08.331681 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:47:08.333926 (+ 2245us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:47:08.337866 (+ 3940us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:47:08.337873 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:47:08.337883 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:47:08.338295 (+ 412us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:47:08.338783 (+ 488us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:47:08.338787 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:47:08.338789 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:47:08.338963 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:47:08.339801 (+ 838us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:47:08.339807 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:47:08.340083 (+ 276us) client_negotiation.cc:769] Sending connection context
0619 16:47:08.340324 (+ 241us) client_negotiation.cc:240] Negotiation successful
0619 16:47:08.340759 (+ 435us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":507,"thread_start_us":310,"threads_started":1}
I20260619 16:47:08.342103 16625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:47:08.320868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:43881 (local address 127.0.97.62:44977)
0619 16:47:08.321590 (+ 722us) server_negotiation.cc:206] Beginning negotiation
0619 16:47:08.321606 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0619 16:47:08.321955 (+ 349us) server_negotiation.cc:407] Connection header received
0619 16:47:08.323921 (+ 1966us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:47:08.323934 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:47:08.324263 (+ 329us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:47:08.324575 (+ 312us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:47:08.326431 (+ 1856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:47:08.327724 (+ 1293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:47:08.329789 (+ 2065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:47:08.330518 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:47:08.330645 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:47:08.334203 (+ 3558us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:47:08.334234 (+ 31us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:47:08.334252 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:47:08.334295 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:47:08.337709 (+ 3414us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:47:08.338427 (+ 718us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:47:08.338436 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:47:08.338443 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:47:08.338648 (+ 205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:47:08.339068 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:47:08.339072 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:47:08.339075 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:47:08.339697 (+ 622us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:47:08.339858 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0619 16:47:08.340571 (+ 713us) server_negotiation.cc:299] Negotiation successful
0619 16:47:08.340968 (+ 397us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":484,"thread_start_us":309,"threads_started":1}
I20260619 16:47:08.344973 16623 heartbeater.cc:347] Connected to a master server at 127.0.97.62:44977
I20260619 16:47:08.345247 16623 heartbeater.cc:464] Registering TS with master...
I20260619 16:47:08.345950 16623 heartbeater.cc:511] Master 127.0.97.62:44977 requested a full tablet report, sending...
I20260619 16:47:08.348222 16438 ts_manager.cc:194] Registered new tserver with Master: be15a6b9c4824e229834519ec2b988d0 (127.0.97.1:42315)
I20260619 16:47:08.350317 16438 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:43881
W20260619 16:47:08.638695 16627 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:47:08.638952 16627 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:47:08.639041 16627 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:47:08.646147 16627 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:47:08.646248 16627 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:47:08.646364 16627 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260619 16:47:08.655692 16627 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45131
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887385903558-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887385903558-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887385903558-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:34415
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887385903558-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=36797
--tserver_master_addrs=127.0.97.62:44977
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887385903558-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:47:08.656854 16627 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:47:08.658079 16627 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:47:08.670084 16633 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:47:08.670118 16635 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:47:08.670138 16632 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:47:08.670964 16627 server_base.cc:1034] running on GCE node
I20260619 16:47:08.671495 16627 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:47:08.672335 16627 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:47:08.673522 16627 hybrid_clock.cc:648] HybridClock initialized: now 1781887628673477 us; error 35 us; skew 500 ppm
I20260619 16:47:08.673830 16627 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:47:08 dist-test-slave-wvz7 krb5kdc[15854](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887628, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:47:08.676672 16627 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260619 16:47:08.678031 16627 webserver.cc:460] Webserver started at http://127.0.97.2:36797/ using document root <none> and password file <none>
I20260619 16:47:08.678653 16627 fs_manager.cc:359] Metadata directory not provided
I20260619 16:47:08.678772 16627 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:47:08.683806 16627 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.003s
I20260619 16:47:08.686302 16642 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:47:08.687484 16627 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 16:47:08.687606 16627 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887385903558-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887385903558-388-0/minicluster-data/ts-1/wal
uuid: "8b323dc8ce0f412db10a79763cb324ce"
format_stamp: "Formatted at 2026-06-19 16:46:46 on dist-test-slave-wvz7"
server_key: "e57f6b3f703bb844abffa382690ab4e7"
server_key_iv: "ffb1bf17e0f01f46f7035afa583b8e56"
server_key_version: "encryptionkey@0"
I20260619 16:47:08.688512 16627 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887385903558-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887385903558-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887385903558-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:47:08.715229 16627 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:47:08.719087 16627 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:47:08.719327 16627 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:47:08.720252 16627 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:47:08.721485 16627 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:47:08.721581 16627 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:47:08.721676 16627 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:47:08.721761 16627 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:47:08.750943 16627 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:34415
I20260619 16:47:08.751016 16754 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:34415 every 8 connection(s)
I20260619 16:47:08.752349 16627 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887385903558-388-0/minicluster-data/ts-1/data/info.pb
I20260619 16:47:08.759775 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu as pid 16627
I20260619 16:47:08.760360 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu
/tmp/dist-test-taskEs2EH8/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887385903558-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887385903558-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887385903558-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887385903558-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:37569
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:44977
--webserver_port=37241
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887385903558-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45131
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887385903558-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887385903558-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887385903558-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 16:47:08 dist-test-slave-wvz7 krb5kdc[15854](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887628, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 16:47:08.772083 16625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:47:08.754723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:50875 (local address 127.0.97.62:44977)
0619 16:47:08.754923 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0619 16:47:08.754928 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:47:08.756185 (+ 1257us) server_negotiation.cc:407] Connection header received
0619 16:47:08.757965 (+ 1780us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:47:08.757970 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:47:08.758037 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:47:08.758124 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:47:08.759693 (+ 1569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:47:08.760419 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:47:08.762076 (+ 1657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:47:08.762664 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:47:08.762790 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:47:08.766458 (+ 3668us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:47:08.766464 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:47:08.766469 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:47:08.766511 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:47:08.768924 (+ 2413us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:47:08.769866 (+ 942us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:47:08.769871 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:47:08.769873 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:47:08.769997 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:47:08.770441 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:47:08.770446 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:47:08.770450 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:47:08.770762 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:47:08.770876 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0619 16:47:08.771648 (+ 772us) server_negotiation.cc:299] Negotiation successful
0619 16:47:08.771841 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260619 16:47:08.773537 16757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:47:08.755293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44977 (local address 127.0.97.2:50875)
0619 16:47:08.755999 (+ 706us) negotiation.cc:107] Waiting for socket to connect
0619 16:47:08.756054 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0619 16:47:08.757669 (+ 1615us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:47:08.758374 (+ 705us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:47:08.758383 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:47:08.758897 (+ 514us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:47:08.759489 (+ 592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:47:08.759507 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:47:08.760620 (+ 1113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:47:08.760624 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:47:08.761926 (+ 1302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:47:08.761940 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:47:08.762928 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:47:08.762936 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:47:08.763178 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:47:08.763835 (+ 657us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:47:08.763859 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:47:08.766199 (+ 2340us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:47:08.769076 (+ 2877us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:47:08.769085 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:47:08.769103 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:47:08.769729 (+ 626us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:47:08.770089 (+ 360us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:47:08.770093 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:47:08.770096 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:47:08.770331 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:47:08.770879 (+ 548us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:47:08.770886 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:47:08.771303 (+ 417us) client_negotiation.cc:769] Sending connection context
0619 16:47:08.771585 (+ 282us) client_negotiation.cc:240] Negotiation successful
0619 16:47:08.772152 (+ 567us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":466,"thread_start_us":293,"threads_started":1}
I20260619 16:47:08.775036 16755 heartbeater.cc:347] Connected to a master server at 127.0.97.62:44977
I20260619 16:47:08.775347 16755 heartbeater.cc:464] Registering TS with master...
I20260619 16:47:08.776163 16755 heartbeater.cc:511] Master 127.0.97.62:44977 requested a full tablet report, sending...
I20260619 16:47:08.777695 16438 ts_manager.cc:194] Registered new tserver with Master: 8b323dc8ce0f412db10a79763cb324ce (127.0.97.2:34415)
I20260619 16:47:08.778548 16438 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:50875
W20260619 16:47:09.074316 16758 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:47:09.074667 16758 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:47:09.074815 16758 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:47:09.082680 16758 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:47:09.082803 16758 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:47:09.082965 16758 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260619 16:47:09.092360 16758 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45131
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887385903558-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887385903558-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887385903558-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:37569
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887385903558-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=37241
--tserver_master_addrs=127.0.97.62:44977
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887385903558-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:47:09.093609 16758 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:47:09.094868 16758 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:47:09.107148 16763 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:47:09.107196 16764 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:47:09.107326 16766 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:47:09.108094 16758 server_base.cc:1034] running on GCE node
I20260619 16:47:09.108680 16758 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:47:09.109571 16758 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:47:09.110759 16758 hybrid_clock.cc:648] HybridClock initialized: now 1781887629110700 us; error 56 us; skew 500 ppm
I20260619 16:47:09.111091 16758 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:47:09 dist-test-slave-wvz7 krb5kdc[15854](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887629, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:47:09.114037 16758 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260619 16:47:09.115414 16758 webserver.cc:460] Webserver started at http://127.0.97.3:37241/ using document root <none> and password file <none>
I20260619 16:47:09.116083 16758 fs_manager.cc:359] Metadata directory not provided
I20260619 16:47:09.116209 16758 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:47:09.121320 16758 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260619 16:47:09.123930 16773 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:47:09.125226 16758 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260619 16:47:09.125366 16758 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887385903558-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887385903558-388-0/minicluster-data/ts-2/wal
uuid: "1efaebd5eb7a4bba9df40e4b0093546c"
format_stamp: "Formatted at 2026-06-19 16:46:46 on dist-test-slave-wvz7"
server_key: "926aba1dd5bd0febc4ab69191b110e40"
server_key_iv: "236335f03fb312342966ceaf411779d1"
server_key_version: "encryptionkey@0"
I20260619 16:47:09.126323 16758 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887385903558-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887385903558-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887385903558-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:47:09.155923 16758 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:47:09.160045 16758 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:47:09.160290 16758 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:47:09.161231 16758 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:47:09.162534 16758 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:47:09.162626 16758 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:47:09.162734 16758 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:47:09.162806 16758 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:47:09.192436 16758 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:37569
I20260619 16:47:09.192600 16885 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:37569 every 8 connection(s)
I20260619 16:47:09.193920 16758 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887385903558-388-0/minicluster-data/ts-2/data/info.pb
I20260619 16:47:09.194696 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu as pid 16758
Jun 19 16:47:09 dist-test-slave-wvz7 krb5kdc[15854](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887629, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 16:47:09.209800 16625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:47:09.195788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47134 (local address 127.0.97.62:44977)
0619 16:47:09.195992 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0619 16:47:09.195999 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:47:09.196769 (+ 770us) server_negotiation.cc:407] Connection header received
0619 16:47:09.196838 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:47:09.196841 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:47:09.196905 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:47:09.196990 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:47:09.198624 (+ 1634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:47:09.199359 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:47:09.200881 (+ 1522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:47:09.201302 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:47:09.201407 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:47:09.203945 (+ 2538us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:47:09.203949 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:47:09.203953 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:47:09.203982 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:47:09.207201 (+ 3219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:47:09.207839 (+ 638us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:47:09.207845 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:47:09.207849 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:47:09.207969 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:47:09.208309 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:47:09.208314 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:47:09.208318 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:47:09.208597 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:47:09.208842 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0619 16:47:09.209131 (+ 289us) server_negotiation.cc:299] Negotiation successful
0619 16:47:09.209401 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260619 16:47:09.214897 16889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:47:09.196572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:40419 (local address 127.0.97.62:44977)
0619 16:47:09.197041 (+ 469us) server_negotiation.cc:206] Beginning negotiation
0619 16:47:09.197047 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:47:09.198128 (+ 1081us) server_negotiation.cc:407] Connection header received
0619 16:47:09.199973 (+ 1845us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:47:09.199979 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:47:09.200071 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:47:09.200178 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:47:09.201618 (+ 1440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:47:09.202525 (+ 907us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:47:09.204019 (+ 1494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:47:09.204518 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:47:09.204628 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:47:09.208468 (+ 3840us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:47:09.208473 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:47:09.208477 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:47:09.208517 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:47:09.211463 (+ 2946us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:47:09.212354 (+ 891us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:47:09.212360 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:47:09.212364 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:47:09.212508 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:47:09.212954 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:47:09.212959 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:47:09.212963 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:47:09.213259 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:47:09.213401 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0619 16:47:09.214185 (+ 784us) server_negotiation.cc:299] Negotiation successful
0619 16:47:09.214430 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":370,"thread_start_us":521,"threads_started":1}
I20260619 16:47:09.215860 16890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:47:09.197173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44977 (local address 127.0.97.3:40419)
0619 16:47:09.197974 (+ 801us) negotiation.cc:107] Waiting for socket to connect
0619 16:47:09.198022 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0619 16:47:09.199738 (+ 1716us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:47:09.200358 (+ 620us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:47:09.200365 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:47:09.200902 (+ 537us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:47:09.201447 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:47:09.201463 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:47:09.202673 (+ 1210us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:47:09.202677 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:47:09.203881 (+ 1204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:47:09.203893 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:47:09.204704 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:47:09.204711 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:47:09.204992 (+ 281us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:47:09.205589 (+ 597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:47:09.205614 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:47:09.208216 (+ 2602us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:47:09.211591 (+ 3375us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:47:09.211600 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:47:09.211618 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:47:09.212244 (+ 626us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:47:09.212620 (+ 376us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:47:09.212626 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:47:09.212631 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:47:09.212837 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:47:09.213390 (+ 553us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:47:09.213398 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:47:09.213794 (+ 396us) client_negotiation.cc:769] Sending connection context
0619 16:47:09.214087 (+ 293us) client_negotiation.cc:240] Negotiation successful
0619 16:47:09.214647 (+ 560us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":509,"thread_start_us":324,"threads_started":1}
I20260619 16:47:09.217252 16886 heartbeater.cc:347] Connected to a master server at 127.0.97.62:44977
I20260619 16:47:09.217541 16886 heartbeater.cc:464] Registering TS with master...
I20260619 16:47:09.218288 16886 heartbeater.cc:511] Master 127.0.97.62:44977 requested a full tablet report, sending...
I20260619 16:47:09.219659 16438 ts_manager.cc:194] Registered new tserver with Master: 1efaebd5eb7a4bba9df40e4b0093546c (127.0.97.3:37569)
I20260619 16:47:09.220461 16438 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:40419
I20260619 16:47:09.224704 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 16:47:09.231189 16889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:47:09.225554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47140 (local address 127.0.97.62:44977)
0619 16:47:09.225789 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0619 16:47:09.225796 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:47:09.226158 (+ 362us) server_negotiation.cc:407] Connection header received
0619 16:47:09.226389 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:47:09.226393 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:47:09.226585 (+ 192us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:47:09.226728 (+ 143us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 16:47:09.227084 (+ 356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:47:09.227849 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:47:09.229372 (+ 1523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:47:09.229870 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:47:09.229963 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:47:09.230438 (+ 475us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 16:47:09.230656 (+ 218us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0619 16:47:09.230931 (+ 275us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:47140: token expired
Metrics: {"server-negotiator.queue_time_us":66}
W20260619 16:47:09.231318 16889 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:47140: token expired
W20260619 16:47:09.231792 16361 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.0.97.62:44977)
I20260619 16:47:09.243975 16889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:47:09.232233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47154 (local address 127.0.97.62:44977)
0619 16:47:09.232382 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0619 16:47:09.232386 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:47:09.232400 (+ 14us) server_negotiation.cc:407] Connection header received
0619 16:47:09.232499 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:47:09.232504 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:47:09.232563 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:47:09.232637 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:47:09.234024 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:47:09.234760 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:47:09.236047 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:47:09.236460 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:47:09.236560 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:47:09.239020 (+ 2460us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:47:09.239026 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:47:09.239029 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:47:09.239063 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:47:09.241504 (+ 2441us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:47:09.242203 (+ 699us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:47:09.242207 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:47:09.242210 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:47:09.242341 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:47:09.242700 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:47:09.242704 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:47:09.242706 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:47:09.243086 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:47:09.243275 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0619 16:47:09.243550 (+ 275us) server_negotiation.cc:299] Negotiation successful
0619 16:47:09.243705 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260619 16:47:09.247331 16438 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47154:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 16:47:09.254676 16438 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 16:47:09.276996 16899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:47:09.266786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50042 (local address 127.0.97.1:42315)
0619 16:47:09.269867 (+ 3081us) server_negotiation.cc:206] Beginning negotiation
0619 16:47:09.269873 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:47:09.269897 (+ 24us) server_negotiation.cc:407] Connection header received
0619 16:47:09.269989 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:47:09.269996 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:47:09.270234 (+ 238us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:47:09.271068 (+ 834us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:47:09.271279 (+ 211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:47:09.272503 (+ 1224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:47:09.274432 (+ 1929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:47:09.275823 (+ 1391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:47:09.276229 (+ 406us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:47:09.276428 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0619 16:47:09.276496 (+ 68us) server_negotiation.cc:299] Negotiation successful
0619 16:47:09.276673 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2921,"thread_start_us":2821,"threads_started":1}
I20260619 16:47:09.277069 16898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:47:09.266364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48788 (local address 127.0.97.3:37569)
0619 16:47:09.268636 (+ 2272us) server_negotiation.cc:206] Beginning negotiation
0619 16:47:09.268646 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 16:47:09.268681 (+ 35us) server_negotiation.cc:407] Connection header received
0619 16:47:09.268788 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:47:09.268796 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:47:09.269073 (+ 277us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:47:09.269263 (+ 190us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:47:09.269977 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:47:09.271985 (+ 2008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:47:09.274862 (+ 2877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:47:09.275771 (+ 909us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:47:09.275890 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:47:09.276123 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0619 16:47:09.276593 (+ 470us) server_negotiation.cc:299] Negotiation successful
0619 16:47:09.276756 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2094,"thread_start_us":1871,"threads_started":1}
I20260619 16:47:09.277069 16896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:47:09.266259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42315 (local address 127.0.0.1:50042)
0619 16:47:09.267118 (+ 859us) negotiation.cc:107] Waiting for socket to connect
0619 16:47:09.267162 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0619 16:47:09.267317 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:47:09.270475 (+ 3158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:47:09.270479 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:47:09.270504 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:47:09.270870 (+ 366us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:47:09.270882 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:47:09.272637 (+ 1755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:47:09.272640 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:47:09.274232 (+ 1592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:47:09.274247 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:47:09.276064 (+ 1817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:47:09.276071 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:47:09.276315 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:47:09.276340 (+ 25us) client_negotiation.cc:769] Sending connection context
0619 16:47:09.276433 (+ 93us) client_negotiation.cc:240] Negotiation successful
0619 16:47:09.276642 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":604,"mutex_wait_us":27,"thread_start_us":653,"threads_started":1}
I20260619 16:47:09.277617 16900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:47:09.268013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:34415 (local address 127.0.0.1:60368)
0619 16:47:09.268786 (+ 773us) negotiation.cc:107] Waiting for socket to connect
0619 16:47:09.268807 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 16:47:09.268917 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:47:09.270037 (+ 1120us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:47:09.270041 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:47:09.270065 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:47:09.270395 (+ 330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:47:09.270407 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:47:09.271924 (+ 1517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:47:09.271930 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:47:09.274659 (+ 2729us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:47:09.274677 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:47:09.276616 (+ 1939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:47:09.276623 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:47:09.276783 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:47:09.276816 (+ 33us) client_negotiation.cc:769] Sending connection context
0619 16:47:09.276880 (+ 64us) client_negotiation.cc:240] Negotiation successful
0619 16:47:09.277008 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":660,"mutex_wait_us":73,"thread_start_us":720,"threads_started":1}
I20260619 16:47:09.278045 16901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:47:09.268596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60368 (local address 127.0.97.2:34415)
0619 16:47:09.269426 (+ 830us) server_negotiation.cc:206] Beginning negotiation
0619 16:47:09.269434 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 16:47:09.269466 (+ 32us) server_negotiation.cc:407] Connection header received
0619 16:47:09.269555 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:47:09.269563 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:47:09.269869 (+ 306us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:47:09.270056 (+ 187us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:47:09.270602 (+ 546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:47:09.271798 (+ 1196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:47:09.274987 (+ 3189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:47:09.276384 (+ 1397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:47:09.277347 (+ 963us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:47:09.277529 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0619 16:47:09.277611 (+ 82us) server_negotiation.cc:299] Negotiation successful
0619 16:47:09.277788 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":631,"thread_start_us":515,"threads_started":1}
I20260619 16:47:09.278340 16897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:47:09.266294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:37569 (local address 127.0.0.1:48788)
0619 16:47:09.267844 (+ 1550us) negotiation.cc:107] Waiting for socket to connect
0619 16:47:09.267859 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 16:47:09.267973 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:47:09.269386 (+ 1413us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:47:09.269394 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:47:09.269425 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:47:09.269801 (+ 376us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:47:09.269814 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:47:09.272147 (+ 2333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:47:09.272152 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:47:09.274720 (+ 2568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:47:09.274731 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:47:09.275984 (+ 1253us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:47:09.275990 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:47:09.276261 (+ 271us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:47:09.276318 (+ 57us) client_negotiation.cc:769] Sending connection context
0619 16:47:09.276437 (+ 119us) client_negotiation.cc:240] Negotiation successful
0619 16:47:09.278009 (+ 1572us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1429,"mutex_wait_us":1260,"thread_start_us":1296,"threads_started":1}
I20260619 16:47:09.281008 16690 tablet_service.cc:1467] Processing CreateTablet for tablet b3e48dd34c004503af174b9e5958a3ef (DEFAULT_TABLE table=test-table [id=cef0900e40f34bdd96c2f5c70c7dc1a8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:47:09.281195 16558 tablet_service.cc:1467] Processing CreateTablet for tablet b3e48dd34c004503af174b9e5958a3ef (DEFAULT_TABLE table=test-table [id=cef0900e40f34bdd96c2f5c70c7dc1a8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:47:09.281391 16821 tablet_service.cc:1467] Processing CreateTablet for tablet b3e48dd34c004503af174b9e5958a3ef (DEFAULT_TABLE table=test-table [id=cef0900e40f34bdd96c2f5c70c7dc1a8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:47:09.282167 16690 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b3e48dd34c004503af174b9e5958a3ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:47:09.282495 16558 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b3e48dd34c004503af174b9e5958a3ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:47:09.282522 16821 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b3e48dd34c004503af174b9e5958a3ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:47:09.293382 16902 tablet_bootstrap.cc:492] T b3e48dd34c004503af174b9e5958a3ef P 8b323dc8ce0f412db10a79763cb324ce: Bootstrap starting.
I20260619 16:47:09.296919 16903 tablet_bootstrap.cc:492] T b3e48dd34c004503af174b9e5958a3ef P be15a6b9c4824e229834519ec2b988d0: Bootstrap starting.
I20260619 16:47:09.297331 16902 tablet_bootstrap.cc:654] T b3e48dd34c004503af174b9e5958a3ef P 8b323dc8ce0f412db10a79763cb324ce: Neither blocks nor log segments found. Creating new log.
I20260619 16:47:09.297689 16904 tablet_bootstrap.cc:492] T b3e48dd34c004503af174b9e5958a3ef P 1efaebd5eb7a4bba9df40e4b0093546c: Bootstrap starting.
I20260619 16:47:09.298660 16902 log.cc:824] T b3e48dd34c004503af174b9e5958a3ef P 8b323dc8ce0f412db10a79763cb324ce: Log is configured to *not* fsync() on all Append() calls
I20260619 16:47:09.301036 16902 tablet_bootstrap.cc:492] T b3e48dd34c004503af174b9e5958a3ef P 8b323dc8ce0f412db10a79763cb324ce: No bootstrap required, opened a new log
I20260619 16:47:09.301023 16904 tablet_bootstrap.cc:654] T b3e48dd34c004503af174b9e5958a3ef P 1efaebd5eb7a4bba9df40e4b0093546c: Neither blocks nor log segments found. Creating new log.
I20260619 16:47:09.301312 16902 ts_tablet_manager.cc:1397] T b3e48dd34c004503af174b9e5958a3ef P 8b323dc8ce0f412db10a79763cb324ce: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260619 16:47:09.302361 16904 log.cc:824] T b3e48dd34c004503af174b9e5958a3ef P 1efaebd5eb7a4bba9df40e4b0093546c: Log is configured to *not* fsync() on all Append() calls
I20260619 16:47:09.302472 16903 tablet_bootstrap.cc:654] T b3e48dd34c004503af174b9e5958a3ef P be15a6b9c4824e229834519ec2b988d0: Neither blocks nor log segments found. Creating new log.
I20260619 16:47:09.303480 16903 log.cc:824] T b3e48dd34c004503af174b9e5958a3ef P be15a6b9c4824e229834519ec2b988d0: Log is configured to *not* fsync() on all Append() calls
I20260619 16:47:09.305078 16904 tablet_bootstrap.cc:492] T b3e48dd34c004503af174b9e5958a3ef P 1efaebd5eb7a4bba9df40e4b0093546c: No bootstrap required, opened a new log
I20260619 16:47:09.305318 16904 ts_tablet_manager.cc:1397] T b3e48dd34c004503af174b9e5958a3ef P 1efaebd5eb7a4bba9df40e4b0093546c: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260619 16:47:09.306324 16903 tablet_bootstrap.cc:492] T b3e48dd34c004503af174b9e5958a3ef P be15a6b9c4824e229834519ec2b988d0: No bootstrap required, opened a new log
I20260619 16:47:09.306571 16903 ts_tablet_manager.cc:1397] T b3e48dd34c004503af174b9e5958a3ef P be15a6b9c4824e229834519ec2b988d0: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.000s
I20260619 16:47:09.310923 16902 raft_consensus.cc:348] T b3e48dd34c004503af174b9e5958a3ef P 8b323dc8ce0f412db10a79763cb324ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1efaebd5eb7a4bba9df40e4b0093546c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37569 } } peers { permanent_uuid: "be15a6b9c4824e229834519ec2b988d0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42315 } } peers { permanent_uuid: "8b323dc8ce0f412db10a79763cb324ce" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34415 } }
I20260619 16:47:09.311349 16902 raft_consensus.cc:374] T b3e48dd34c004503af174b9e5958a3ef P 8b323dc8ce0f412db10a79763cb324ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:47:09.311445 16902 raft_consensus.cc:729] T b3e48dd34c004503af174b9e5958a3ef P 8b323dc8ce0f412db10a79763cb324ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b323dc8ce0f412db10a79763cb324ce, State: Initialized, Role: FOLLOWER
I20260619 16:47:09.312043 16902 consensus_queue.cc:260] T b3e48dd34c004503af174b9e5958a3ef P 8b323dc8ce0f412db10a79763cb324ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1efaebd5eb7a4bba9df40e4b0093546c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37569 } } peers { permanent_uuid: "be15a6b9c4824e229834519ec2b988d0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42315 } } peers { permanent_uuid: "8b323dc8ce0f412db10a79763cb324ce" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34415 } }
I20260619 16:47:09.312080 16904 raft_consensus.cc:348] T b3e48dd34c004503af174b9e5958a3ef P 1efaebd5eb7a4bba9df40e4b0093546c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1efaebd5eb7a4bba9df40e4b0093546c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37569 } } peers { permanent_uuid: "be15a6b9c4824e229834519ec2b988d0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42315 } } peers { permanent_uuid: "8b323dc8ce0f412db10a79763cb324ce" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34415 } }
I20260619 16:47:09.312372 16904 raft_consensus.cc:374] T b3e48dd34c004503af174b9e5958a3ef P 1efaebd5eb7a4bba9df40e4b0093546c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:47:09.312454 16904 raft_consensus.cc:729] T b3e48dd34c004503af174b9e5958a3ef P 1efaebd5eb7a4bba9df40e4b0093546c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1efaebd5eb7a4bba9df40e4b0093546c, State: Initialized, Role: FOLLOWER
I20260619 16:47:09.313174 16755 heartbeater.cc:503] Master 127.0.97.62:44977 was elected leader, sending a full tablet report...
I20260619 16:47:09.313078 16904 consensus_queue.cc:260] T b3e48dd34c004503af174b9e5958a3ef P 1efaebd5eb7a4bba9df40e4b0093546c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1efaebd5eb7a4bba9df40e4b0093546c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37569 } } peers { permanent_uuid: "be15a6b9c4824e229834519ec2b988d0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42315 } } peers { permanent_uuid: "8b323dc8ce0f412db10a79763cb324ce" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34415 } }
I20260619 16:47:09.313751 16902 ts_tablet_manager.cc:1428] T b3e48dd34c004503af174b9e5958a3ef P 8b323dc8ce0f412db10a79763cb324ce: Time spent starting tablet: real 0.012s user 0.009s sys 0.004s
I20260619 16:47:09.315649 16886 heartbeater.cc:503] Master 127.0.97.62:44977 was elected leader, sending a full tablet report...
I20260619 16:47:09.315966 16904 ts_tablet_manager.cc:1428] T b3e48dd34c004503af174b9e5958a3ef P 1efaebd5eb7a4bba9df40e4b0093546c: Time spent starting tablet: real 0.011s user 0.009s sys 0.002s
I20260619 16:47:09.316002 16903 raft_consensus.cc:348] T b3e48dd34c004503af174b9e5958a3ef P be15a6b9c4824e229834519ec2b988d0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1efaebd5eb7a4bba9df40e4b0093546c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37569 } } peers { permanent_uuid: "be15a6b9c4824e229834519ec2b988d0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42315 } } peers { permanent_uuid: "8b323dc8ce0f412db10a79763cb324ce" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34415 } }
I20260619 16:47:09.316552 16903 raft_consensus.cc:374] T b3e48dd34c004503af174b9e5958a3ef P be15a6b9c4824e229834519ec2b988d0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:47:09.316692 16903 raft_consensus.cc:729] T b3e48dd34c004503af174b9e5958a3ef P be15a6b9c4824e229834519ec2b988d0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: be15a6b9c4824e229834519ec2b988d0, State: Initialized, Role: FOLLOWER
I20260619 16:47:09.317270 16903 consensus_queue.cc:260] T b3e48dd34c004503af174b9e5958a3ef P be15a6b9c4824e229834519ec2b988d0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1efaebd5eb7a4bba9df40e4b0093546c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37569 } } peers { permanent_uuid: "be15a6b9c4824e229834519ec2b988d0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42315 } } peers { permanent_uuid: "8b323dc8ce0f412db10a79763cb324ce" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34415 } }
I20260619 16:47:09.319442 16623 heartbeater.cc:503] Master 127.0.97.62:44977 was elected leader, sending a full tablet report...
I20260619 16:47:09.320147 16903 ts_tablet_manager.cc:1428] T b3e48dd34c004503af174b9e5958a3ef P be15a6b9c4824e229834519ec2b988d0: Time spent starting tablet: real 0.013s user 0.011s sys 0.003s
W20260619 16:47:09.320344 16624 tablet.cc:2369] T b3e48dd34c004503af174b9e5958a3ef P be15a6b9c4824e229834519ec2b988d0: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 16:47:09.445660 16887 tablet.cc:2369] T b3e48dd34c004503af174b9e5958a3ef P 1efaebd5eb7a4bba9df40e4b0093546c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 16:47:09.491653 16910 raft_consensus.cc:482] T b3e48dd34c004503af174b9e5958a3ef P be15a6b9c4824e229834519ec2b988d0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 16:47:09.491971 16910 raft_consensus.cc:504] T b3e48dd34c004503af174b9e5958a3ef P be15a6b9c4824e229834519ec2b988d0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1efaebd5eb7a4bba9df40e4b0093546c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37569 } } peers { permanent_uuid: "be15a6b9c4824e229834519ec2b988d0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42315 } } peers { permanent_uuid: "8b323dc8ce0f412db10a79763cb324ce" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34415 } }
I20260619 16:47:09.493724 16910 leader_election.cc:288] T b3e48dd34c004503af174b9e5958a3ef P be15a6b9c4824e229834519ec2b988d0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1efaebd5eb7a4bba9df40e4b0093546c (127.0.97.3:37569), 8b323dc8ce0f412db10a79763cb324ce (127.0.97.2:34415)
I20260619 16:47:09.499922 16898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:47:09.494197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:37211 (local address 127.0.97.3:37569)
0619 16:47:09.494383 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0619 16:47:09.494388 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:47:09.494743 (+ 355us) server_negotiation.cc:407] Connection header received
0619 16:47:09.494953 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:47:09.494959 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:47:09.495037 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:47:09.495141 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:47:09.495745 (+ 604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:47:09.496503 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:47:09.498275 (+ 1772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:47:09.499050 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:47:09.499165 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:47:09.499267 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0619 16:47:09.499544 (+ 277us) server_negotiation.cc:299] Negotiation successful
0619 16:47:09.499680 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260619 16:47:09.500031 16911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:47:09.494065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:37569 (local address 127.0.97.1:37211)
0619 16:47:09.494653 (+ 588us) negotiation.cc:107] Waiting for socket to connect
0619 16:47:09.494673 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 16:47:09.494813 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:47:09.495272 (+ 459us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:47:09.495280 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:47:09.495309 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:47:09.495588 (+ 279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:47:09.495599 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:47:09.496647 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:47:09.496652 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:47:09.498129 (+ 1477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:47:09.498141 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:47:09.499244 (+ 1103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:47:09.499251 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:47:09.499402 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:47:09.499425 (+ 23us) client_negotiation.cc:769] Sending connection context
0619 16:47:09.499506 (+ 81us) client_negotiation.cc:240] Negotiation successful
0619 16:47:09.499634 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":460,"mutex_wait_us":47,"thread_start_us":287,"threads_started":1}
I20260619 16:47:09.500334 16901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:47:09.494942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:49067 (local address 127.0.97.2:34415)
0619 16:47:09.495179 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0619 16:47:09.495185 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:47:09.495535 (+ 350us) server_negotiation.cc:407] Connection header received
0619 16:47:09.495701 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:47:09.495705 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:47:09.495759 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:47:09.495829 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:47:09.496377 (+ 548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:47:09.497022 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:47:09.498684 (+ 1662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:47:09.499434 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:47:09.499549 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:47:09.499647 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0619 16:47:09.499966 (+ 319us) server_negotiation.cc:299] Negotiation successful
0619 16:47:09.500090 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260619 16:47:09.500425 16912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:47:09.494835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:34415 (local address 127.0.97.1:49067)
0619 16:47:09.495433 (+ 598us) negotiation.cc:107] Waiting for socket to connect
0619 16:47:09.495447 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 16:47:09.495576 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:47:09.495852 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:47:09.495856 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:47:09.495878 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:47:09.496219 (+ 341us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:47:09.496240 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:47:09.497138 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:47:09.497144 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:47:09.498544 (+ 1400us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:47:09.498556 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:47:09.499635 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:47:09.499642 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:47:09.499824 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:47:09.499848 (+ 24us) client_negotiation.cc:769] Sending connection context
0619 16:47:09.499932 (+ 84us) client_negotiation.cc:240] Negotiation successful
0619 16:47:09.500053 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":487,"mutex_wait_us":1,"thread_start_us":291,"threads_started":1}
I20260619 16:47:09.501101 16841 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3e48dd34c004503af174b9e5958a3ef" candidate_uuid: "be15a6b9c4824e229834519ec2b988d0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1efaebd5eb7a4bba9df40e4b0093546c" is_pre_election: true
I20260619 16:47:09.501224 16710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3e48dd34c004503af174b9e5958a3ef" candidate_uuid: "be15a6b9c4824e229834519ec2b988d0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b323dc8ce0f412db10a79763cb324ce" is_pre_election: true
I20260619 16:47:09.501581 16841 raft_consensus.cc:2454] T b3e48dd34c004503af174b9e5958a3ef P 1efaebd5eb7a4bba9df40e4b0093546c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate be15a6b9c4824e229834519ec2b988d0 in term 0.
I20260619 16:47:09.501655 16710 raft_consensus.cc:2454] T b3e48dd34c004503af174b9e5958a3ef P 8b323dc8ce0f412db10a79763cb324ce [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate be15a6b9c4824e229834519ec2b988d0 in term 0.
I20260619 16:47:09.502228 16513 leader_election.cc:302] T b3e48dd34c004503af174b9e5958a3ef P be15a6b9c4824e229834519ec2b988d0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1efaebd5eb7a4bba9df40e4b0093546c, be15a6b9c4824e229834519ec2b988d0; no voters:
I20260619 16:47:09.502609 16910 raft_consensus.cc:2781] T b3e48dd34c004503af174b9e5958a3ef P be15a6b9c4824e229834519ec2b988d0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 16:47:09.502746 16910 raft_consensus.cc:482] T b3e48dd34c004503af174b9e5958a3ef P be15a6b9c4824e229834519ec2b988d0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 16:47:09.502849 16910 raft_consensus.cc:3037] T b3e48dd34c004503af174b9e5958a3ef P be15a6b9c4824e229834519ec2b988d0 [term 0 FOLLOWER]: Advancing to term 1
W20260619 16:47:09.504177 16756 tablet.cc:2369] T b3e48dd34c004503af174b9e5958a3ef P 8b323dc8ce0f412db10a79763cb324ce: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 16:47:09.505496 16910 raft_consensus.cc:504] T b3e48dd34c004503af174b9e5958a3ef P be15a6b9c4824e229834519ec2b988d0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1efaebd5eb7a4bba9df40e4b0093546c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37569 } } peers { permanent_uuid: "be15a6b9c4824e229834519ec2b988d0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42315 } } peers { permanent_uuid: "8b323dc8ce0f412db10a79763cb324ce" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34415 } }
I20260619 16:47:09.506141 16910 leader_election.cc:288] T b3e48dd34c004503af174b9e5958a3ef P be15a6b9c4824e229834519ec2b988d0 [CANDIDATE]: Term 1 election: Requested vote from peers 1efaebd5eb7a4bba9df40e4b0093546c (127.0.97.3:37569), 8b323dc8ce0f412db10a79763cb324ce (127.0.97.2:34415)
I20260619 16:47:09.506637 16841 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3e48dd34c004503af174b9e5958a3ef" candidate_uuid: "be15a6b9c4824e229834519ec2b988d0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1efaebd5eb7a4bba9df40e4b0093546c"
I20260619 16:47:09.506668 16710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3e48dd34c004503af174b9e5958a3ef" candidate_uuid: "be15a6b9c4824e229834519ec2b988d0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b323dc8ce0f412db10a79763cb324ce"
I20260619 16:47:09.506861 16710 raft_consensus.cc:3037] T b3e48dd34c004503af174b9e5958a3ef P 8b323dc8ce0f412db10a79763cb324ce [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:47:09.506861 16841 raft_consensus.cc:3037] T b3e48dd34c004503af174b9e5958a3ef P 1efaebd5eb7a4bba9df40e4b0093546c [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:47:09.509939 16710 raft_consensus.cc:2454] T b3e48dd34c004503af174b9e5958a3ef P 8b323dc8ce0f412db10a79763cb324ce [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate be15a6b9c4824e229834519ec2b988d0 in term 1.
I20260619 16:47:09.509939 16841 raft_consensus.cc:2454] T b3e48dd34c004503af174b9e5958a3ef P 1efaebd5eb7a4bba9df40e4b0093546c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate be15a6b9c4824e229834519ec2b988d0 in term 1.
I20260619 16:47:09.510382 16513 leader_election.cc:302] T b3e48dd34c004503af174b9e5958a3ef P be15a6b9c4824e229834519ec2b988d0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1efaebd5eb7a4bba9df40e4b0093546c, be15a6b9c4824e229834519ec2b988d0; no voters:
I20260619 16:47:09.510658 16910 raft_consensus.cc:2781] T b3e48dd34c004503af174b9e5958a3ef P be15a6b9c4824e229834519ec2b988d0 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:47:09.511085 16910 raft_consensus.cc:686] T b3e48dd34c004503af174b9e5958a3ef P be15a6b9c4824e229834519ec2b988d0 [term 1 LEADER]: Becoming Leader. State: Replica: be15a6b9c4824e229834519ec2b988d0, State: Running, Role: LEADER
I20260619 16:47:09.511536 16910 consensus_queue.cc:237] T b3e48dd34c004503af174b9e5958a3ef P be15a6b9c4824e229834519ec2b988d0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1efaebd5eb7a4bba9df40e4b0093546c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37569 } } peers { permanent_uuid: "be15a6b9c4824e229834519ec2b988d0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42315 } } peers { permanent_uuid: "8b323dc8ce0f412db10a79763cb324ce" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34415 } }
I20260619 16:47:09.515784 16436 catalog_manager.cc:5495] T b3e48dd34c004503af174b9e5958a3ef P be15a6b9c4824e229834519ec2b988d0 reported cstate change: term changed from 0 to 1, leader changed from <none> to be15a6b9c4824e229834519ec2b988d0 (127.0.97.1). New cstate: current_term: 1 leader_uuid: "be15a6b9c4824e229834519ec2b988d0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1efaebd5eb7a4bba9df40e4b0093546c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37569 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "be15a6b9c4824e229834519ec2b988d0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42315 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8b323dc8ce0f412db10a79763cb324ce" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34415 } health_report { overall_health: UNKNOWN } } }
I20260619 16:47:09.574663 16899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:47:09.569406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50056 (local address 127.0.97.1:42315)
0619 16:47:09.569595 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0619 16:47:09.569601 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:47:09.569763 (+ 162us) server_negotiation.cc:407] Connection header received
0619 16:47:09.569995 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:47:09.569999 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:47:09.570079 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:47:09.570208 (+ 129us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 16:47:09.570565 (+ 357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:47:09.571263 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:47:09.572589 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:47:09.572995 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:47:09.573103 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:47:09.573582 (+ 479us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 16:47:09.574087 (+ 505us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 16:47:09.574229 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0619 16:47:09.574310 (+ 81us) server_negotiation.cc:299] Negotiation successful
0619 16:47:09.574392 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260619 16:47:09.582645 16841 raft_consensus.cc:1261] T b3e48dd34c004503af174b9e5958a3ef P 1efaebd5eb7a4bba9df40e4b0093546c [term 1 FOLLOWER]: Refusing update from remote peer be15a6b9c4824e229834519ec2b988d0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:47:09.582657 16710 raft_consensus.cc:1261] T b3e48dd34c004503af174b9e5958a3ef P 8b323dc8ce0f412db10a79763cb324ce [term 1 FOLLOWER]: Refusing update from remote peer be15a6b9c4824e229834519ec2b988d0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:47:09.583593 16910 consensus_queue.cc:1035] T b3e48dd34c004503af174b9e5958a3ef P be15a6b9c4824e229834519ec2b988d0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b323dc8ce0f412db10a79763cb324ce" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34415 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:47:09.583907 16913 consensus_queue.cc:1035] T b3e48dd34c004503af174b9e5958a3ef P be15a6b9c4824e229834519ec2b988d0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1efaebd5eb7a4bba9df40e4b0093546c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37569 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:47:09.613425 16436 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47154:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:47:09.613709 16436 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47154:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:47:09.617136 16436 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 460891f6ca034e7c99713cb56b72d30f: Sending DeleteTablet for 3 replicas of tablet b3e48dd34c004503af174b9e5958a3ef
I20260619 16:47:09.618301 16558 tablet_service.cc:1514] Processing DeleteTablet for tablet b3e48dd34c004503af174b9e5958a3ef with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:47:09 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:50042
I20260619 16:47:09.618438 16821 tablet_service.cc:1514] Processing DeleteTablet for tablet b3e48dd34c004503af174b9e5958a3ef with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:47:09 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:48788
I20260619 16:47:09.618438 16690 tablet_service.cc:1514] Processing DeleteTablet for tablet b3e48dd34c004503af174b9e5958a3ef with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:47:09 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:60368
I20260619 16:47:09.619630 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu pid 16475
I20260619 16:47:09.629418 16927 tablet_replica.cc:330] T b3e48dd34c004503af174b9e5958a3ef P 8b323dc8ce0f412db10a79763cb324ce: stopping tablet replica
I20260619 16:47:09.629551 16926 tablet_replica.cc:330] T b3e48dd34c004503af174b9e5958a3ef P 1efaebd5eb7a4bba9df40e4b0093546c: stopping tablet replica
I20260619 16:47:09.629926 16927 raft_consensus.cc:2229] T b3e48dd34c004503af174b9e5958a3ef P 8b323dc8ce0f412db10a79763cb324ce [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 16:47:09.630012 16926 raft_consensus.cc:2229] T b3e48dd34c004503af174b9e5958a3ef P 1efaebd5eb7a4bba9df40e4b0093546c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 16:47:09.630489 16927 raft_consensus.cc:2258] T b3e48dd34c004503af174b9e5958a3ef P 8b323dc8ce0f412db10a79763cb324ce [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:47:09.630498 16926 raft_consensus.cc:2258] T b3e48dd34c004503af174b9e5958a3ef P 1efaebd5eb7a4bba9df40e4b0093546c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:47:09.632905 16926 ts_tablet_manager.cc:1905] T b3e48dd34c004503af174b9e5958a3ef P 1efaebd5eb7a4bba9df40e4b0093546c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:47:09.638500 16899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:47:09.620554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50064 (local address 127.0.97.1:42315)
0619 16:47:09.620759 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0619 16:47:09.620767 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 16:47:09.620795 (+ 28us) server_negotiation.cc:407] Connection header received
0619 16:47:09.620889 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:47:09.620896 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:47:09.621011 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:47:09.621133 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:47:09.622771 (+ 1638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:47:09.624220 (+ 1449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:47:09.626171 (+ 1951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:47:09.626984 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:47:09.627142 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:47:09.630507 (+ 3365us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:47:09.630523 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:47:09.630540 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:47:09.630686 (+ 146us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:47:09.635159 (+ 4473us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:47:09.635982 (+ 823us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:47:09.635992 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:47:09.635996 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:47:09.636261 (+ 265us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:47:09.636685 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:47:09.636691 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:47:09.636694 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:47:09.637193 (+ 499us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:47:09.637529 (+ 336us) server_negotiation.cc:1035] Waiting for connection context
0619 16:47:09.637834 (+ 305us) server_negotiation.cc:299] Negotiation successful
0619 16:47:09.638078 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260619 16:47:09.639642 16618 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:47:09.641040 16927 ts_tablet_manager.cc:1905] T b3e48dd34c004503af174b9e5958a3ef P 8b323dc8ce0f412db10a79763cb324ce: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:47:09.641522 16926 ts_tablet_manager.cc:1918] T b3e48dd34c004503af174b9e5958a3ef P 1efaebd5eb7a4bba9df40e4b0093546c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 16:47:09.641752 16926 log.cc:1196] T b3e48dd34c004503af174b9e5958a3ef P 1efaebd5eb7a4bba9df40e4b0093546c: Deleting WAL directory at /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887385903558-388-0/minicluster-data/ts-2/wal/wals/b3e48dd34c004503af174b9e5958a3ef
I20260619 16:47:09.642351 16926 ts_tablet_manager.cc:1939] T b3e48dd34c004503af174b9e5958a3ef P 1efaebd5eb7a4bba9df40e4b0093546c: Deleting consensus metadata
I20260619 16:47:09.644505 16424 catalog_manager.cc:4841] TS 1efaebd5eb7a4bba9df40e4b0093546c (127.0.97.3:37569): tablet b3e48dd34c004503af174b9e5958a3ef (table test-table [id=cef0900e40f34bdd96c2f5c70c7dc1a8]) successfully deleted
I20260619 16:47:09.648150 16927 ts_tablet_manager.cc:1918] T b3e48dd34c004503af174b9e5958a3ef P 8b323dc8ce0f412db10a79763cb324ce: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 16:47:09.648340 16927 log.cc:1196] T b3e48dd34c004503af174b9e5958a3ef P 8b323dc8ce0f412db10a79763cb324ce: Deleting WAL directory at /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887385903558-388-0/minicluster-data/ts-1/wal/wals/b3e48dd34c004503af174b9e5958a3ef
I20260619 16:47:09.648815 16927 ts_tablet_manager.cc:1939] T b3e48dd34c004503af174b9e5958a3ef P 8b323dc8ce0f412db10a79763cb324ce: Deleting consensus metadata
I20260619 16:47:09.650496 16424 catalog_manager.cc:4841] TS 8b323dc8ce0f412db10a79763cb324ce (127.0.97.2:34415): tablet b3e48dd34c004503af174b9e5958a3ef (table test-table [id=cef0900e40f34bdd96c2f5c70c7dc1a8]) successfully deleted
I20260619 16:47:09.739528 16925 tablet_replica.cc:330] T b3e48dd34c004503af174b9e5958a3ef P be15a6b9c4824e229834519ec2b988d0: stopping tablet replica
I20260619 16:47:09.739974 16925 raft_consensus.cc:2229] T b3e48dd34c004503af174b9e5958a3ef P be15a6b9c4824e229834519ec2b988d0 [term 1 LEADER]: Raft consensus shutting down.
I20260619 16:47:09.740624 16925 raft_consensus.cc:2258] T b3e48dd34c004503af174b9e5958a3ef P be15a6b9c4824e229834519ec2b988d0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:47:09.741535 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu with pid 16475
I20260619 16:47:09.756101 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu pid 16627
W20260619 16:47:09.756102 16422 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.97.1:42315 (error 108)
I20260619 16:47:09.758052 16897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:47:09.757618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42315 (local address 127.0.0.1:50076)
0619 16:47:09.757811 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0619 16:47:09.757927 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:42315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260619 16:47:09.758275 16422 catalog_manager.cc:4584] TS be15a6b9c4824e229834519ec2b988d0 (127.0.97.1:42315): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b3e48dd34c004503af174b9e5958a3ef: Network error: Client connection negotiation failed: client connection to 127.0.97.1:42315: connect: Connection refused (error 111)
I20260619 16:47:09.772292 16901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:47:09.757320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60380 (local address 127.0.97.2:34415)
0619 16:47:09.757496 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0619 16:47:09.757500 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:47:09.757514 (+ 14us) server_negotiation.cc:407] Connection header received
0619 16:47:09.757561 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:47:09.757565 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:47:09.757649 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:47:09.757767 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:47:09.759506 (+ 1739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:47:09.760313 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:47:09.762417 (+ 2104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:47:09.762916 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:47:09.763017 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:47:09.766346 (+ 3329us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:47:09.766358 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:47:09.766373 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:47:09.766403 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:47:09.769301 (+ 2898us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:47:09.770179 (+ 878us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:47:09.770190 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:47:09.770193 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:47:09.770370 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:47:09.770804 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:47:09.770808 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:47:09.770810 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:47:09.771191 (+ 381us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:47:09.771411 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0619 16:47:09.771747 (+ 336us) server_negotiation.cc:299] Negotiation successful
0619 16:47:09.771970 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260619 16:47:09.772945 16750 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:47:09.811791 16897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:47:09.811248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42315 (local address 127.0.0.1:50080)
0619 16:47:09.811478 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0619 16:47:09.811611 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:42315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260619 16:47:09.812844 16897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:47:09.812441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42315 (local address 127.0.0.1:50084)
0619 16:47:09.812595 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0619 16:47:09.812695 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:42315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260619 16:47:09.860679 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu with pid 16627
I20260619 16:47:09.873510 16897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:47:09.872926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42315 (local address 127.0.0.1:50098)
0619 16:47:09.873179 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0619 16:47:09.873291 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:42315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260619 16:47:09.874691 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu pid 16758
I20260619 16:47:09.874747 16897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:47:09.874351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42315 (local address 127.0.0.1:50106)
0619 16:47:09.874572 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0619 16:47:09.874638 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:42315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113}
I20260619 16:47:09.888319 16898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:47:09.875425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48792 (local address 127.0.97.3:37569)
0619 16:47:09.875584 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0619 16:47:09.875589 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:47:09.875609 (+ 20us) server_negotiation.cc:407] Connection header received
0619 16:47:09.875787 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:47:09.875791 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:47:09.875855 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:47:09.875955 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:47:09.877246 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:47:09.877998 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:47:09.879249 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:47:09.879628 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:47:09.879737 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:47:09.882353 (+ 2616us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:47:09.882368 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:47:09.882384 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:47:09.882417 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:47:09.885225 (+ 2808us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:47:09.886134 (+ 909us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:47:09.886145 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:47:09.886149 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:47:09.886331 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:47:09.886705 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:47:09.886709 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:47:09.886713 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:47:09.887160 (+ 447us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:47:09.887451 (+ 291us) server_negotiation.cc:1035] Waiting for connection context
0619 16:47:09.887740 (+ 289us) server_negotiation.cc:299] Negotiation successful
0619 16:47:09.887961 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260619 16:47:09.888927 16881 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:47:09.955761 16897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:47:09.955071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42315 (local address 127.0.0.1:50120)
0619 16:47:09.955393 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0619 16:47:09.955529 (+ 136us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:42315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119}
I20260619 16:47:09.956974 16897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:47:09.956553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42315 (local address 127.0.0.1:50132)
0619 16:47:09.956773 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0619 16:47:09.956849 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:42315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124}
I20260619 16:47:09.978058 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu with pid 16758
I20260619 16:47:09.990782 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu pid 16405
I20260619 16:47:09.991902 16468 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:47:10.104915 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu with pid 16405
I20260619 16:47:30.122232 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu
/tmp/dist-test-taskEs2EH8/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887385903558-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887385903558-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887385903558-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887385903558-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:44977
--webserver_interface=127.0.97.62
--webserver_port=43343
--keytab_file=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887385903558-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45131
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:44977
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887385903558-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887385903558-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887385903558-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:47:30.437711 16933 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:47:30.438057 16933 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:47:30.438184 16933 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:47:30.445472 16933 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260619 16:47:30.445591 16933 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260619 16:47:30.445654 16933 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260619 16:47:30.445724 16933 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:47:30.445816 16933 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:47:30.445904 16933 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:47:30.445963 16933 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:47:30.446043 16933 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:47:30.455809 16933 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45131
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887385903558-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887385903558-388-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:44977
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887385903558-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:44977
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887385903558-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=43343
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887385903558-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:47:30.457058 16933 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:47:30.458360 16933 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:47:30.469823 16938 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:47:30.469720 16941 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:47:30.469794 16939 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:47:30.470057 16933 server_base.cc:1034] running on GCE node
I20260619 16:47:30.471118 16933 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:47:30.472602 16933 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:47:30.473865 16933 hybrid_clock.cc:648] HybridClock initialized: now 1781887650473794 us; error 38 us; skew 500 ppm
I20260619 16:47:30.474238 16933 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:47:30 dist-test-slave-wvz7 krb5kdc[15854](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887650, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:47:30.480305 16933 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260619 16:47:30.481827 16933 webserver.cc:460] Webserver started at http://127.0.97.62:43343/ using document root <none> and password file <none>
I20260619 16:47:30.482527 16933 fs_manager.cc:359] Metadata directory not provided
I20260619 16:47:30.482625 16933 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:47:30.488023 16933 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 16:47:30.491817 16948 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:47:30.493261 16933 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 16:47:30.493428 16933 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887385903558-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887385903558-388-0/minicluster-data/master-0/wal
uuid: "460891f6ca034e7c99713cb56b72d30f"
format_stamp: "Formatted at 2026-06-19 16:46:45 on dist-test-slave-wvz7"
server_key: "45c073fb68fba1d7d41be8c7cb093973"
server_key_iv: "5428244b0fcbb52060183807b9da9d45"
server_key_version: "encryptionkey@0"
I20260619 16:47:30.494468 16933 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887385903558-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887385903558-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887385903558-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:47:30.514479 16933 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:47:30.518642 16933 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:47:30.518903 16933 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:47:30.536299 16933 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:44977
I20260619 16:47:30.536339 16999 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:44977 every 8 connection(s)
I20260619 16:47:30.537856 16933 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887385903558-388-0/minicluster-data/master-0/data/info.pb
I20260619 16:47:30.543376 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu as pid 16933
I20260619 16:47:30.544162 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu
/tmp/dist-test-taskEs2EH8/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887385903558-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887385903558-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887385903558-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887385903558-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:42315
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:44977
--webserver_port=43223
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887385903558-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45131
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887385903558-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887385903558-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887385903558-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 16:47:30.544726 17000 sys_catalog.cc:263] Verifying existing consensus state
I20260619 16:47:30.547916 17000 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 460891f6ca034e7c99713cb56b72d30f: Bootstrap starting.
I20260619 16:47:30.574609 17000 log.cc:824] T 00000000000000000000000000000000 P 460891f6ca034e7c99713cb56b72d30f: Log is configured to *not* fsync() on all Append() calls
I20260619 16:47:30.595544 17000 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 460891f6ca034e7c99713cb56b72d30f: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 16:47:30.596220 17000 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 460891f6ca034e7c99713cb56b72d30f: Bootstrap complete.
I20260619 16:47:30.610566 17000 raft_consensus.cc:348] T 00000000000000000000000000000000 P 460891f6ca034e7c99713cb56b72d30f [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "460891f6ca034e7c99713cb56b72d30f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44977 } }
I20260619 16:47:30.612100 17000 raft_consensus.cc:729] T 00000000000000000000000000000000 P 460891f6ca034e7c99713cb56b72d30f [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 460891f6ca034e7c99713cb56b72d30f, State: Initialized, Role: FOLLOWER
I20260619 16:47:30.613006 17000 consensus_queue.cc:260] T 00000000000000000000000000000000 P 460891f6ca034e7c99713cb56b72d30f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "460891f6ca034e7c99713cb56b72d30f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44977 } }
I20260619 16:47:30.613338 17000 raft_consensus.cc:388] T 00000000000000000000000000000000 P 460891f6ca034e7c99713cb56b72d30f [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:47:30.613492 17000 raft_consensus.cc:482] T 00000000000000000000000000000000 P 460891f6ca034e7c99713cb56b72d30f [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:47:30.613698 17000 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 460891f6ca034e7c99713cb56b72d30f [term 2 FOLLOWER]: Advancing to term 3
I20260619 16:47:30.619700 17000 raft_consensus.cc:504] T 00000000000000000000000000000000 P 460891f6ca034e7c99713cb56b72d30f [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "460891f6ca034e7c99713cb56b72d30f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44977 } }
I20260619 16:47:30.620352 17000 leader_election.cc:302] T 00000000000000000000000000000000 P 460891f6ca034e7c99713cb56b72d30f [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 460891f6ca034e7c99713cb56b72d30f; no voters:
I20260619 16:47:30.621084 17000 leader_election.cc:288] T 00000000000000000000000000000000 P 460891f6ca034e7c99713cb56b72d30f [CANDIDATE]: Term 3 election: Requested vote from peers
I20260619 16:47:30.621243 17004 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 460891f6ca034e7c99713cb56b72d30f [term 3 FOLLOWER]: Leader election won for term 3
I20260619 16:47:30.622367 17004 raft_consensus.cc:686] T 00000000000000000000000000000000 P 460891f6ca034e7c99713cb56b72d30f [term 3 LEADER]: Becoming Leader. State: Replica: 460891f6ca034e7c99713cb56b72d30f, State: Running, Role: LEADER
I20260619 16:47:30.622972 17004 consensus_queue.cc:237] T 00000000000000000000000000000000 P 460891f6ca034e7c99713cb56b72d30f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "460891f6ca034e7c99713cb56b72d30f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44977 } }
I20260619 16:47:30.623279 17000 sys_catalog.cc:564] T 00000000000000000000000000000000 P 460891f6ca034e7c99713cb56b72d30f [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:47:30.625376 17006 sys_catalog.cc:455] T 00000000000000000000000000000000 P 460891f6ca034e7c99713cb56b72d30f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 460891f6ca034e7c99713cb56b72d30f. Latest consensus state: current_term: 3 leader_uuid: "460891f6ca034e7c99713cb56b72d30f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "460891f6ca034e7c99713cb56b72d30f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44977 } } }
I20260619 16:47:30.625981 17006 sys_catalog.cc:458] T 00000000000000000000000000000000 P 460891f6ca034e7c99713cb56b72d30f [sys.catalog]: This master's current role is: LEADER
I20260619 16:47:30.625631 17005 sys_catalog.cc:455] T 00000000000000000000000000000000 P 460891f6ca034e7c99713cb56b72d30f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "460891f6ca034e7c99713cb56b72d30f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "460891f6ca034e7c99713cb56b72d30f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44977 } } }
I20260619 16:47:30.626818 17005 sys_catalog.cc:458] T 00000000000000000000000000000000 P 460891f6ca034e7c99713cb56b72d30f [sys.catalog]: This master's current role is: LEADER
I20260619 16:47:30.635193 17010 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:47:30.644416 17010 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:47:30.646082 17010 catalog_manager.cc:1259] Loaded cluster ID: 5075d572c7964b97b3497f85f2f1032f
I20260619 16:47:30.646247 17010 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:47:30.650992 17010 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:47:30.652941 17010 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 460891f6ca034e7c99713cb56b72d30f: Loaded TSK: 0
I20260619 16:47:30.653182 17010 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 460891f6ca034e7c99713cb56b72d30f: Loaded TSK: 1
F20260619 16:47:30.654419 17010 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f6c72fff9ad google::LogMessage::Fail() at ??:0
@ 0x7f6c7300288a google::LogMessage::SendToLog() at ??:0
@ 0x7f6c72fff4de google::LogMessage::Flush() at ??:0
@ 0x7f6c73000a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f6c76d637d2 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f6c76d5273e kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f6c88a2e1c0 kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f6c88a2d783 kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f6c88ab92a7 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_10::operator()() at ??:0
I20260619 16:47:30.690336 17024 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
@ 0x7f6c88ab90c4 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x7f6c88b035e9 std::function<>::operator()() at ??:0
@ 0x7f6c88a31cdd kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7f6c88a2fc36 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f6c88ab7610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7f6c88ab7439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f6c75c4b345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f6c75c4e91c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f6c75c4e799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f6c75c29eaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7f6c78f036db start_thread at ??:0
@ 0x7f6c71e9961f clone at ??:0
W20260619 16:47:30.941612 17002 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:47:30.941955 17002 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:47:30.942106 17002 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:47:30.949651 17002 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:47:30.949811 17002 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:47:30.949980 17002 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260619 16:47:30.960172 17002 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45131
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887385903558-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887385903558-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887385903558-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:42315
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887385903558-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=43223
--tserver_master_addrs=127.0.97.62:44977
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887385903558-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:47:30.961448 17002 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:47:30.962780 17002 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:47:30.975221 17047 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:47:30.975322 17048 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:47:30.975288 17050 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:47:30.976181 17002 server_base.cc:1034] running on GCE node
I20260619 16:47:30.976733 17002 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:47:30.977579 17002 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:47:30.978792 17002 hybrid_clock.cc:648] HybridClock initialized: now 1781887650978727 us; error 46 us; skew 500 ppm
I20260619 16:47:30.979069 17002 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:47:30 dist-test-slave-wvz7 krb5kdc[15854](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887650, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:47:30.982028 17002 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260619 16:47:30.983327 17002 webserver.cc:460] Webserver started at http://127.0.97.1:43223/ using document root <none> and password file <none>
I20260619 16:47:30.983958 17002 fs_manager.cc:359] Metadata directory not provided
I20260619 16:47:30.984061 17002 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:47:30.989013 17002 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260619 16:47:30.991758 17057 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:47:30.993180 17002 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 16:47:30.993333 17002 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887385903558-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887385903558-388-0/minicluster-data/ts-0/wal
uuid: "be15a6b9c4824e229834519ec2b988d0"
format_stamp: "Formatted at 2026-06-19 16:46:45 on dist-test-slave-wvz7"
server_key: "3c2545cdbeb192463e0a296228190a40"
server_key_iv: "8d196395b19860edfa33881c321b1719"
server_key_version: "encryptionkey@0"
I20260619 16:47:30.994364 17002 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887385903558-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887385903558-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887385903558-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:47:31.011022 17002 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:47:31.015107 17002 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:47:31.015355 17002 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:47:31.016340 17002 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:47:31.018172 17064 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20260619 16:47:31.023617 17002 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260619 16:47:31.023729 17002 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.000s sys 0.000s
I20260619 16:47:31.023849 17002 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260619 16:47:31.026895 17002 ts_tablet_manager.cc:610] Registered 1 tablets
I20260619 16:47:31.026966 17002 ts_tablet_manager.cc:589] Time spent register tablets: real 0.003s user 0.002s sys 0.000s
I20260619 16:47:31.027376 17064 tablet_bootstrap.cc:492] T b3e48dd34c004503af174b9e5958a3ef P be15a6b9c4824e229834519ec2b988d0: Bootstrap starting.
I20260619 16:47:31.054734 17064 log.cc:824] T b3e48dd34c004503af174b9e5958a3ef P be15a6b9c4824e229834519ec2b988d0: Log is configured to *not* fsync() on all Append() calls
I20260619 16:47:31.066375 17002 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:42315
I20260619 16:47:31.066401 17171 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:42315 every 8 connection(s)
I20260619 16:47:31.067603 17002 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887385903558-388-0/minicluster-data/ts-0/data/info.pb
I20260619 16:47:31.068509 17064 tablet_bootstrap.cc:492] T b3e48dd34c004503af174b9e5958a3ef P be15a6b9c4824e229834519ec2b988d0: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 16:47:31.069482 17064 tablet_bootstrap.cc:492] T b3e48dd34c004503af174b9e5958a3ef P be15a6b9c4824e229834519ec2b988d0: Bootstrap complete.
I20260619 16:47:31.070688 17064 ts_tablet_manager.cc:1397] T b3e48dd34c004503af174b9e5958a3ef P be15a6b9c4824e229834519ec2b988d0: Time spent bootstrapping tablet: real 0.044s user 0.029s sys 0.013s
I20260619 16:47:31.071460 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu as pid 17002
I20260619 16:47:31.072125 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu
/tmp/dist-test-taskEs2EH8/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887385903558-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887385903558-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887385903558-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887385903558-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:34415
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:44977
--webserver_port=36797
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887385903558-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45131
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887385903558-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887385903558-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887385903558-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 16:47:31.072983 17174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:47:31.070472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44977 (local address 127.0.97.1:55819)
0619 16:47:31.071477 (+ 1005us) negotiation.cc:107] Waiting for socket to connect
0619 16:47:31.071742 (+ 265us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:44977: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":699,"thread_start_us":460,"threads_started":1}
W20260619 16:47:31.073835 17172 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:44977 (0 consecutive failures): Network error: Failed to ping master at 127.0.97.62:44977: Client connection negotiation failed: client connection to 127.0.97.62:44977: connect: Connection refused (error 111)
I20260619 16:47:31.074887 17174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:47:31.074452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44977 (local address 127.0.97.1:51279)
0619 16:47:31.074625 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0619 16:47:31.074723 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:44977: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260619 16:47:31.075934 17174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:47:31.075499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44977 (local address 127.0.97.1:56713)
0619 16:47:31.075648 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0619 16:47:31.075762 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:44977: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260619 16:47:31.076249 17172 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260619 16:47:31.079221 17064 raft_consensus.cc:348] T b3e48dd34c004503af174b9e5958a3ef P be15a6b9c4824e229834519ec2b988d0 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1efaebd5eb7a4bba9df40e4b0093546c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37569 } } peers { permanent_uuid: "be15a6b9c4824e229834519ec2b988d0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42315 } } peers { permanent_uuid: "8b323dc8ce0f412db10a79763cb324ce" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34415 } }
I20260619 16:47:31.080152 17064 raft_consensus.cc:729] T b3e48dd34c004503af174b9e5958a3ef P be15a6b9c4824e229834519ec2b988d0 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: be15a6b9c4824e229834519ec2b988d0, State: Initialized, Role: FOLLOWER
I20260619 16:47:31.080773 17064 consensus_queue.cc:260] T b3e48dd34c004503af174b9e5958a3ef P be15a6b9c4824e229834519ec2b988d0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1efaebd5eb7a4bba9df40e4b0093546c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37569 } } peers { permanent_uuid: "be15a6b9c4824e229834519ec2b988d0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42315 } } peers { permanent_uuid: "8b323dc8ce0f412db10a79763cb324ce" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34415 } }
I20260619 16:47:31.082410 17174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:47:31.082055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44977 (local address 127.0.97.1:51269)
0619 16:47:31.082201 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0619 16:47:31.082277 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:44977: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260619 16:47:31.082433 17064 ts_tablet_manager.cc:1428] T b3e48dd34c004503af174b9e5958a3ef P be15a6b9c4824e229834519ec2b988d0: Time spent starting tablet: real 0.012s user 0.008s sys 0.004s
I20260619 16:47:31.083372 17174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:47:31.083062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44977 (local address 127.0.97.1:49431)
0619 16:47:31.083184 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0619 16:47:31.083251 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:44977: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260619 16:47:31.384915 17175 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:47:31.385176 17175 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:47:31.385268 17175 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:47:31.392168 17175 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:47:31.392264 17175 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:47:31.392378 17175 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260619 16:47:31.401808 17175 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45131
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887385903558-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887385903558-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887385903558-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:34415
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887385903558-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=36797
--tserver_master_addrs=127.0.97.62:44977
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887385903558-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:47:31.402927 17175 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:47:31.404397 17175 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:47:31.417073 17181 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:47:31.417110 17184 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:47:31.417137 17182 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:47:31.418511 17175 server_base.cc:1034] running on GCE node
I20260619 16:47:31.419129 17175 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:47:31.420150 17175 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:47:31.421365 17175 hybrid_clock.cc:648] HybridClock initialized: now 1781887651421315 us; error 44 us; skew 500 ppm
I20260619 16:47:31.421802 17175 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:47:31 dist-test-slave-wvz7 krb5kdc[15854](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887651, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:47:31.424896 17175 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260619 16:47:31.426466 17175 webserver.cc:460] Webserver started at http://127.0.97.2:36797/ using document root <none> and password file <none>
I20260619 16:47:31.427126 17175 fs_manager.cc:359] Metadata directory not provided
I20260619 16:47:31.427225 17175 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:47:31.432482 17175 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 16:47:31.435312 17191 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:47:31.436661 17175 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 16:47:31.436798 17175 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887385903558-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887385903558-388-0/minicluster-data/ts-1/wal
uuid: "8b323dc8ce0f412db10a79763cb324ce"
format_stamp: "Formatted at 2026-06-19 16:46:46 on dist-test-slave-wvz7"
server_key: "e57f6b3f703bb844abffa382690ab4e7"
server_key_iv: "ffb1bf17e0f01f46f7035afa583b8e56"
server_key_version: "encryptionkey@0"
I20260619 16:47:31.437808 17175 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887385903558-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887385903558-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887385903558-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:47:31.454583 17175 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:47:31.458774 17175 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:47:31.459028 17175 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:47:31.459996 17175 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:47:31.461300 17175 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:47:31.461386 17175 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:47:31.461488 17175 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:47:31.461561 17175 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:47:31.492828 17175 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:34415
I20260619 16:47:31.492880 17303 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:34415 every 8 connection(s)
I20260619 16:47:31.494241 17175 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887385903558-388-0/minicluster-data/ts-1/data/info.pb
I20260619 16:47:31.495355 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu as pid 17175
I20260619 16:47:31.496106 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu
/tmp/dist-test-taskEs2EH8/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887385903558-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887385903558-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887385903558-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887385903558-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:37569
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:44977
--webserver_port=37241
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887385903558-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45131
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887385903558-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887385903558-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887385903558-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 16:47:31.500138 17306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:47:31.497866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44977 (local address 127.0.97.2:42141)
0619 16:47:31.498788 (+ 922us) negotiation.cc:107] Waiting for socket to connect
0619 16:47:31.499029 (+ 241us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:44977: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":532,"thread_start_us":303,"threads_started":1}
W20260619 16:47:31.500954 17304 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:44977 (0 consecutive failures): Network error: Failed to ping master at 127.0.97.62:44977: Client connection negotiation failed: client connection to 127.0.97.62:44977: connect: Connection refused (error 111)
I20260619 16:47:31.501893 17306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:47:31.501501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44977 (local address 127.0.97.2:45967)
0619 16:47:31.501668 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0619 16:47:31.501766 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:44977: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260619 16:47:31.502744 17306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:47:31.502414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44977 (local address 127.0.97.2:53959)
0619 16:47:31.502542 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0619 16:47:31.502627 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:44977: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260619 16:47:31.502982 17304 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260619 16:47:31.808095 17307 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:47:31.808370 17307 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:47:31.808509 17307 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:47:31.815523 17307 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:47:31.815635 17307 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:47:31.815800 17307 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260619 16:47:31.825063 17307 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45131
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887385903558-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887385903558-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887385903558-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:37569
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887385903558-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=37241
--tserver_master_addrs=127.0.97.62:44977
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887385903558-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:47:31.826252 17307 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:47:31.827435 17307 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:47:31.839470 17313 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:47:31.839534 17315 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:47:31.839504 17312 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:47:31.839879 17307 server_base.cc:1034] running on GCE node
I20260619 16:47:31.840678 17307 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:47:31.841627 17307 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:47:31.842813 17307 hybrid_clock.cc:648] HybridClock initialized: now 1781887651842724 us; error 82 us; skew 500 ppm
I20260619 16:47:31.843133 17307 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:47:31 dist-test-slave-wvz7 krb5kdc[15854](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887651, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:47:31.846081 17307 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260619 16:47:31.847534 17307 webserver.cc:460] Webserver started at http://127.0.97.3:37241/ using document root <none> and password file <none>
I20260619 16:47:31.848201 17307 fs_manager.cc:359] Metadata directory not provided
I20260619 16:47:31.848318 17307 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:47:31.853470 17307 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.001s
I20260619 16:47:31.856256 17322 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:47:31.857604 17307 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 16:47:31.857781 17307 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887385903558-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887385903558-388-0/minicluster-data/ts-2/wal
uuid: "1efaebd5eb7a4bba9df40e4b0093546c"
format_stamp: "Formatted at 2026-06-19 16:46:46 on dist-test-slave-wvz7"
server_key: "926aba1dd5bd0febc4ab69191b110e40"
server_key_iv: "236335f03fb312342966ceaf411779d1"
server_key_version: "encryptionkey@0"
I20260619 16:47:31.858721 17307 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887385903558-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887385903558-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887385903558-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:47:31.871373 17307 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:47:31.875519 17307 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:47:31.875795 17307 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:47:31.876856 17307 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:47:31.878232 17307 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:47:31.878328 17307 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:47:31.878435 17307 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:47:31.878501 17307 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:47:31.910141 17307 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:37569
I20260619 16:47:31.910194 17434 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:37569 every 8 connection(s)
I20260619 16:47:31.911527 17307 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887385903558-388-0/minicluster-data/ts-2/data/info.pb
I20260619 16:47:31.916396 17437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:47:31.914584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44977 (local address 127.0.97.3:41399)
0619 16:47:31.915309 (+ 725us) negotiation.cc:107] Waiting for socket to connect
0619 16:47:31.915503 (+ 194us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:44977: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":474,"thread_start_us":278,"threads_started":1}
W20260619 16:47:31.917001 17435 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:44977 (0 consecutive failures): Network error: Failed to ping master at 127.0.97.62:44977: Client connection negotiation failed: client connection to 127.0.97.62:44977: connect: Connection refused (error 111)
I20260619 16:47:31.917762 17437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:47:31.917432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44977 (local address 127.0.97.3:44701)
0619 16:47:31.917571 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0619 16:47:31.917634 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:44977: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260619 16:47:31.918524 17437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:47:31.918221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44977 (local address 127.0.97.3:43387)
0619 16:47:31.918348 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0619 16:47:31.918416 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:44977: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260619 16:47:31.918731 17435 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260619 16:47:31.919575 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu as pid 17307
I20260619 16:47:32.085057 17439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:47:32.084273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44977 (local address 127.0.97.1:33079)
0619 16:47:32.084732 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0619 16:47:32.084826 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:44977: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":356,"thread_start_us":214,"threads_started":1}
I20260619 16:47:32.504397 17440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:47:32.503555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44977 (local address 127.0.97.2:38521)
0619 16:47:32.504046 (+ 491us) negotiation.cc:107] Waiting for socket to connect
0619 16:47:32.504165 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:44977: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":371,"thread_start_us":214,"threads_started":1}
I20260619 16:47:32.653636 17441 raft_consensus.cc:482] T b3e48dd34c004503af174b9e5958a3ef P be15a6b9c4824e229834519ec2b988d0 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 16:47:32.653930 17441 raft_consensus.cc:504] T b3e48dd34c004503af174b9e5958a3ef P be15a6b9c4824e229834519ec2b988d0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1efaebd5eb7a4bba9df40e4b0093546c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37569 } } peers { permanent_uuid: "be15a6b9c4824e229834519ec2b988d0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42315 } } peers { permanent_uuid: "8b323dc8ce0f412db10a79763cb324ce" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34415 } }
I20260619 16:47:32.655563 17441 leader_election.cc:288] T b3e48dd34c004503af174b9e5958a3ef P be15a6b9c4824e229834519ec2b988d0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1efaebd5eb7a4bba9df40e4b0093546c (127.0.97.3:37569), 8b323dc8ce0f412db10a79763cb324ce (127.0.97.2:34415)
Jun 19 16:47:32 dist-test-slave-wvz7 krb5kdc[15854](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887650, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
Jun 19 16:47:32 dist-test-slave-wvz7 krb5kdc[15854](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887650, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260619 16:47:32.680222 17444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:47:32.656596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:34415 (local address 127.0.97.1:56447)
0619 16:47:32.657128 (+ 532us) negotiation.cc:107] Waiting for socket to connect
0619 16:47:32.657151 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0619 16:47:32.657294 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:47:32.660661 (+ 3367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:47:32.660673 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:47:32.661763 (+ 1090us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:47:32.662942 (+ 1179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:47:32.662971 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:47:32.665371 (+ 2400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:47:32.665378 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:47:32.667709 (+ 2331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:47:32.667727 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:47:32.669116 (+ 1389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:47:32.669128 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:47:32.669489 (+ 361us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:47:32.670359 (+ 870us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:47:32.670388 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:47:32.673470 (+ 3082us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:47:32.676741 (+ 3271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:47:32.676745 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:47:32.676747 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:47:32.677193 (+ 446us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:47:32.677653 (+ 460us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:47:32.677658 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:47:32.677662 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:47:32.677834 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:47:32.678662 (+ 828us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:47:32.678667 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:47:32.678911 (+ 244us) client_negotiation.cc:769] Sending connection context
0619 16:47:32.679204 (+ 293us) client_negotiation.cc:240] Negotiation successful
0619 16:47:32.679696 (+ 492us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":408,"mutex_wait_us":281,"spinlock_wait_cycles":15232,"thread_start_us":221,"threads_started":1}
I20260619 16:47:32.680246 17445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:47:32.656937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:56447 (local address 127.0.97.2:34415)
0619 16:47:32.657608 (+ 671us) server_negotiation.cc:206] Beginning negotiation
0619 16:47:32.657622 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 16:47:32.657663 (+ 41us) server_negotiation.cc:407] Connection header received
0619 16:47:32.659734 (+ 2071us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:47:32.659744 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:47:32.660063 (+ 319us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:47:32.660448 (+ 385us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:47:32.663177 (+ 2729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:47:32.665161 (+ 1984us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:47:32.667866 (+ 2705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:47:32.668886 (+ 1020us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:47:32.669067 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:47:32.673707 (+ 4640us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:47:32.673714 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:47:32.673727 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:47:32.673800 (+ 73us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:47:32.676603 (+ 2803us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:47:32.677328 (+ 725us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:47:32.677336 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:47:32.677343 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:47:32.677545 (+ 202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:47:32.677968 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:47:32.677973 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:47:32.677975 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:47:32.678541 (+ 566us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:47:32.678703 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0619 16:47:32.679448 (+ 745us) server_negotiation.cc:299] Negotiation successful
0619 16:47:32.679845 (+ 397us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":481,"thread_start_us":316,"threads_started":1}
I20260619 16:47:32.680459 17443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:47:32.656036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:48359 (local address 127.0.97.3:37569)
0619 16:47:32.656552 (+ 516us) server_negotiation.cc:206] Beginning negotiation
0619 16:47:32.656567 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 16:47:32.656613 (+ 46us) server_negotiation.cc:407] Connection header received
0619 16:47:32.659505 (+ 2892us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:47:32.659518 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:47:32.660058 (+ 540us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:47:32.660448 (+ 390us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:47:32.663242 (+ 2794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:47:32.665111 (+ 1869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:47:32.667866 (+ 2755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:47:32.668886 (+ 1020us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:47:32.669069 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:47:32.673414 (+ 4345us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:47:32.673425 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:47:32.673444 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:47:32.673489 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:47:32.676249 (+ 2760us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:47:32.677121 (+ 872us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:47:32.677130 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:47:32.677137 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:47:32.677352 (+ 215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:47:32.677852 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:47:32.677856 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:47:32.677859 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:47:32.678470 (+ 611us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:47:32.678639 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0619 16:47:32.679431 (+ 792us) server_negotiation.cc:299] Negotiation successful
0619 16:47:32.680019 (+ 588us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":345,"thread_start_us":253,"threads_started":1}
I20260619 16:47:32.680392 17442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:47:32.655824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:37569 (local address 127.0.97.1:48359)
0619 16:47:32.656388 (+ 564us) negotiation.cc:107] Waiting for socket to connect
0619 16:47:32.656424 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0619 16:47:32.657011 (+ 587us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:47:32.660661 (+ 3650us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:47:32.660673 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:47:32.661763 (+ 1090us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:47:32.662942 (+ 1179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:47:32.662971 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:47:32.665374 (+ 2403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:47:32.665378 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:47:32.667711 (+ 2333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:47:32.667727 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:47:32.669149 (+ 1422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:47:32.669156 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:47:32.669494 (+ 338us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:47:32.670355 (+ 861us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:47:32.670397 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:47:32.673108 (+ 2711us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:47:32.676392 (+ 3284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:47:32.676410 (+ 18us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:47:32.676426 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:47:32.676981 (+ 555us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:47:32.677450 (+ 469us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:47:32.677455 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:47:32.677459 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:47:32.677696 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:47:32.678616 (+ 920us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:47:32.678624 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:47:32.678912 (+ 288us) client_negotiation.cc:769] Sending connection context
0619 16:47:32.679137 (+ 225us) client_negotiation.cc:240] Negotiation successful
0619 16:47:32.679887 (+ 750us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":445,"mutex_wait_us":598,"spinlock_wait_cycles":64896,"thread_start_us":315,"threads_started":1}
W20260619 16:47:32.683964 17060 leader_election.cc:341] T b3e48dd34c004503af174b9e5958a3ef P be15a6b9c4824e229834519ec2b988d0 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 8b323dc8ce0f412db10a79763cb324ce (127.0.97.2:34415): Not found: Tablet not found: b3e48dd34c004503af174b9e5958a3ef
W20260619 16:47:32.684651 17060 leader_election.cc:341] T b3e48dd34c004503af174b9e5958a3ef P be15a6b9c4824e229834519ec2b988d0 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1efaebd5eb7a4bba9df40e4b0093546c (127.0.97.3:37569): Not found: Tablet not found: b3e48dd34c004503af174b9e5958a3ef
I20260619 16:47:32.684782 17060 leader_election.cc:302] T b3e48dd34c004503af174b9e5958a3ef P be15a6b9c4824e229834519ec2b988d0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: be15a6b9c4824e229834519ec2b988d0; no voters: 1efaebd5eb7a4bba9df40e4b0093546c, 8b323dc8ce0f412db10a79763cb324ce
I20260619 16:47:32.685166 17441 raft_consensus.cc:2726] T b3e48dd34c004503af174b9e5958a3ef P be15a6b9c4824e229834519ec2b988d0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 16:47:32.920157 17446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:47:32.919260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44977 (local address 127.0.97.3:44657)
0619 16:47:32.919777 (+ 517us) negotiation.cc:107] Waiting for socket to connect
0619 16:47:32.919915 (+ 138us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:44977: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":403,"thread_start_us":262,"threads_started":1}
I20260619 16:47:33.086401 17442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:47:33.085965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44977 (local address 127.0.97.1:37511)
0619 16:47:33.086165 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0619 16:47:33.086261 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:44977: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260619 16:47:33.505971 17447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:47:33.505240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44977 (local address 127.0.97.2:52021)
0619 16:47:33.505670 (+ 430us) negotiation.cc:107] Waiting for socket to connect
0619 16:47:33.505788 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:44977: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":327,"thread_start_us":225,"threads_started":1}
I20260619 16:47:33.921783 17448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:47:33.921060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44977 (local address 127.0.97.3:37297)
0619 16:47:33.921511 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0619 16:47:33.921593 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:44977: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":353,"thread_start_us":239,"threads_started":1}
I20260619 16:47:34.088019 17449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:47:34.087251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44977 (local address 127.0.97.1:50233)
0619 16:47:34.087760 (+ 509us) negotiation.cc:107] Waiting for socket to connect
0619 16:47:34.087840 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:44977: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":407,"thread_start_us":225,"threads_started":1}
I20260619 16:47:34.507519 17450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:47:34.506823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44977 (local address 127.0.97.2:46821)
0619 16:47:34.507250 (+ 427us) negotiation.cc:107] Waiting for socket to connect
0619 16:47:34.507331 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:44977: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":330,"thread_start_us":226,"threads_started":1}
I20260619 16:47:34.825776 17451 raft_consensus.cc:482] T b3e48dd34c004503af174b9e5958a3ef P be15a6b9c4824e229834519ec2b988d0 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 16:47:34.825949 17451 raft_consensus.cc:504] T b3e48dd34c004503af174b9e5958a3ef P be15a6b9c4824e229834519ec2b988d0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1efaebd5eb7a4bba9df40e4b0093546c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37569 } } peers { permanent_uuid: "be15a6b9c4824e229834519ec2b988d0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42315 } } peers { permanent_uuid: "8b323dc8ce0f412db10a79763cb324ce" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34415 } }
I20260619 16:47:34.826550 17451 leader_election.cc:288] T b3e48dd34c004503af174b9e5958a3ef P be15a6b9c4824e229834519ec2b988d0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1efaebd5eb7a4bba9df40e4b0093546c (127.0.97.3:37569), 8b323dc8ce0f412db10a79763cb324ce (127.0.97.2:34415)
W20260619 16:47:34.827557 17060 leader_election.cc:341] T b3e48dd34c004503af174b9e5958a3ef P be15a6b9c4824e229834519ec2b988d0 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 8b323dc8ce0f412db10a79763cb324ce (127.0.97.2:34415): Not found: Tablet not found: b3e48dd34c004503af174b9e5958a3ef
W20260619 16:47:34.827754 17060 leader_election.cc:341] T b3e48dd34c004503af174b9e5958a3ef P be15a6b9c4824e229834519ec2b988d0 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1efaebd5eb7a4bba9df40e4b0093546c (127.0.97.3:37569): Not found: Tablet not found: b3e48dd34c004503af174b9e5958a3ef
I20260619 16:47:34.827829 17060 leader_election.cc:302] T b3e48dd34c004503af174b9e5958a3ef P be15a6b9c4824e229834519ec2b988d0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: be15a6b9c4824e229834519ec2b988d0; no voters: 1efaebd5eb7a4bba9df40e4b0093546c, 8b323dc8ce0f412db10a79763cb324ce
I20260619 16:47:34.828056 17451 raft_consensus.cc:2726] T b3e48dd34c004503af174b9e5958a3ef P be15a6b9c4824e229834519ec2b988d0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 16:47:34.923298 17452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:47:34.922609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44977 (local address 127.0.97.3:52583)
0619 16:47:34.923068 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0619 16:47:34.923146 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:44977: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":357,"thread_start_us":231,"threads_started":1}
I20260619 16:47:35.089861 17453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:47:35.088930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44977 (local address 127.0.97.1:42589)
0619 16:47:35.089470 (+ 540us) negotiation.cc:107] Waiting for socket to connect
0619 16:47:35.089609 (+ 139us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:44977: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":417,"thread_start_us":252,"threads_started":1}
I20260619 16:47:35.509006 17454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:47:35.508328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44977 (local address 127.0.97.2:46903)
0619 16:47:35.508744 (+ 416us) negotiation.cc:107] Waiting for socket to connect
0619 16:47:35.508822 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:44977: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":306,"thread_start_us":197,"threads_started":1}
I20260619 16:47:35.924887 17455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:47:35.924163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44977 (local address 127.0.97.3:60837)
0619 16:47:35.924608 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0619 16:47:35.924690 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:44977: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":334,"thread_start_us":205,"threads_started":1}
I20260619 16:47:36.091557 17456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:47:36.090854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44977 (local address 127.0.97.1:45941)
0619 16:47:36.091295 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0619 16:47:36.091376 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:44977: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":341,"thread_start_us":230,"threads_started":1}
I20260619 16:47:36.402743 17457 raft_consensus.cc:482] T b3e48dd34c004503af174b9e5958a3ef P be15a6b9c4824e229834519ec2b988d0 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 16:47:36.402941 17457 raft_consensus.cc:504] T b3e48dd34c004503af174b9e5958a3ef P be15a6b9c4824e229834519ec2b988d0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1efaebd5eb7a4bba9df40e4b0093546c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37569 } } peers { permanent_uuid: "be15a6b9c4824e229834519ec2b988d0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42315 } } peers { permanent_uuid: "8b323dc8ce0f412db10a79763cb324ce" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34415 } }
I20260619 16:47:36.403618 17457 leader_election.cc:288] T b3e48dd34c004503af174b9e5958a3ef P be15a6b9c4824e229834519ec2b988d0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1efaebd5eb7a4bba9df40e4b0093546c (127.0.97.3:37569), 8b323dc8ce0f412db10a79763cb324ce (127.0.97.2:34415)
W20260619 16:47:36.404598 17060 leader_election.cc:341] T b3e48dd34c004503af174b9e5958a3ef P be15a6b9c4824e229834519ec2b988d0 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1efaebd5eb7a4bba9df40e4b0093546c (127.0.97.3:37569): Not found: Tablet not found: b3e48dd34c004503af174b9e5958a3ef
W20260619 16:47:36.404825 17060 leader_election.cc:341] T b3e48dd34c004503af174b9e5958a3ef P be15a6b9c4824e229834519ec2b988d0 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 8b323dc8ce0f412db10a79763cb324ce (127.0.97.2:34415): Not found: Tablet not found: b3e48dd34c004503af174b9e5958a3ef
I20260619 16:47:36.404937 17060 leader_election.cc:302] T b3e48dd34c004503af174b9e5958a3ef P be15a6b9c4824e229834519ec2b988d0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: be15a6b9c4824e229834519ec2b988d0; no voters: 1efaebd5eb7a4bba9df40e4b0093546c, 8b323dc8ce0f412db10a79763cb324ce
I20260619 16:47:36.405148 17457 raft_consensus.cc:2726] T b3e48dd34c004503af174b9e5958a3ef P be15a6b9c4824e229834519ec2b988d0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 16:47:36.510692 17458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:47:36.509918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44977 (local address 127.0.97.2:48543)
0619 16:47:36.510426 (+ 508us) negotiation.cc:107] Waiting for socket to connect
0619 16:47:36.510517 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:44977: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":404,"thread_start_us":272,"threads_started":1}
I20260619 16:47:36.926437 17459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:47:36.925782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44977 (local address 127.0.97.3:39827)
0619 16:47:36.926194 (+ 412us) negotiation.cc:107] Waiting for socket to connect
0619 16:47:36.926269 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:44977: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":321,"thread_start_us":226,"threads_started":1}
I20260619 16:47:37.093159 17460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:47:37.092467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44977 (local address 127.0.97.1:43715)
0619 16:47:37.092900 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0619 16:47:37.092984 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:44977: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":339,"thread_start_us":229,"threads_started":1}
I20260619 16:47:37.512430 17461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:47:37.511581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44977 (local address 127.0.97.2:36527)
0619 16:47:37.512127 (+ 546us) negotiation.cc:107] Waiting for socket to connect
0619 16:47:37.512219 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:44977: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":437,"thread_start_us":290,"threads_started":1}
I20260619 16:47:37.928169 17462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:47:37.927316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44977 (local address 127.0.97.3:50289)
0619 16:47:37.927892 (+ 576us) negotiation.cc:107] Waiting for socket to connect
0619 16:47:37.927990 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:44977: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":358,"thread_start_us":246,"threads_started":1}
I20260619 16:47:38.094732 17463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:47:38.094021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44977 (local address 127.0.97.1:54113)
0619 16:47:38.094451 (+ 430us) negotiation.cc:107] Waiting for socket to connect
0619 16:47:38.094539 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:44977: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":326,"thread_start_us":224,"threads_started":1}
I20260619 16:47:38.514077 17464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:47:38.513281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44977 (local address 127.0.97.2:36289)
0619 16:47:38.513777 (+ 496us) negotiation.cc:107] Waiting for socket to connect
0619 16:47:38.513866 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:44977: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":365,"thread_start_us":212,"threads_started":1}
I20260619 16:47:38.890574 17465 raft_consensus.cc:482] T b3e48dd34c004503af174b9e5958a3ef P be15a6b9c4824e229834519ec2b988d0 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 16:47:38.890766 17465 raft_consensus.cc:504] T b3e48dd34c004503af174b9e5958a3ef P be15a6b9c4824e229834519ec2b988d0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1efaebd5eb7a4bba9df40e4b0093546c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37569 } } peers { permanent_uuid: "be15a6b9c4824e229834519ec2b988d0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42315 } } peers { permanent_uuid: "8b323dc8ce0f412db10a79763cb324ce" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34415 } }
I20260619 16:47:38.891438 17465 leader_election.cc:288] T b3e48dd34c004503af174b9e5958a3ef P be15a6b9c4824e229834519ec2b988d0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1efaebd5eb7a4bba9df40e4b0093546c (127.0.97.3:37569), 8b323dc8ce0f412db10a79763cb324ce (127.0.97.2:34415)
W20260619 16:47:38.892388 17060 leader_election.cc:341] T b3e48dd34c004503af174b9e5958a3ef P be15a6b9c4824e229834519ec2b988d0 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 8b323dc8ce0f412db10a79763cb324ce (127.0.97.2:34415): Not found: Tablet not found: b3e48dd34c004503af174b9e5958a3ef
W20260619 16:47:38.892642 17060 leader_election.cc:341] T b3e48dd34c004503af174b9e5958a3ef P be15a6b9c4824e229834519ec2b988d0 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1efaebd5eb7a4bba9df40e4b0093546c (127.0.97.3:37569): Not found: Tablet not found: b3e48dd34c004503af174b9e5958a3ef
I20260619 16:47:38.892745 17060 leader_election.cc:302] T b3e48dd34c004503af174b9e5958a3ef P be15a6b9c4824e229834519ec2b988d0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: be15a6b9c4824e229834519ec2b988d0; no voters: 1efaebd5eb7a4bba9df40e4b0093546c, 8b323dc8ce0f412db10a79763cb324ce
I20260619 16:47:38.892998 17465 raft_consensus.cc:2726] T b3e48dd34c004503af174b9e5958a3ef P be15a6b9c4824e229834519ec2b988d0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 16:47:38.929947 17466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:47:38.929131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44977 (local address 127.0.97.3:50009)
0619 16:47:38.929632 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0619 16:47:38.929714 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:44977: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":396,"thread_start_us":222,"threads_started":1}
I20260619 16:47:39.096395 17467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:47:39.095559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44977 (local address 127.0.97.1:36743)
0619 16:47:39.096065 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0619 16:47:39.096178 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:44977: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":398,"thread_start_us":256,"threads_started":1}
I20260619 16:47:39.515720 17468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:47:39.515030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44977 (local address 127.0.97.2:40193)
0619 16:47:39.515446 (+ 416us) negotiation.cc:107] Waiting for socket to connect
0619 16:47:39.515550 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:44977: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":299,"thread_start_us":201,"threads_started":1}
I20260619 16:47:39.931552 17469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:47:39.930824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44977 (local address 127.0.97.3:45461)
0619 16:47:39.931282 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0619 16:47:39.931368 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:44977: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":359,"thread_start_us":255,"threads_started":1}
I20260619 16:47:40.098059 17470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:47:40.097306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44977 (local address 127.0.97.1:57725)
0619 16:47:40.097776 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0619 16:47:40.097859 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:44977: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":350,"thread_start_us":215,"threads_started":1}
I20260619 16:47:40.517308 17471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:47:40.516632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44977 (local address 127.0.97.2:33365)
0619 16:47:40.517069 (+ 437us) negotiation.cc:107] Waiting for socket to connect
0619 16:47:40.517155 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:44977: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":336,"thread_start_us":204,"threads_started":1}
I20260619 16:47:40.933182 17472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:47:40.932386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44977 (local address 127.0.97.3:52809)
0619 16:47:40.932942 (+ 556us) negotiation.cc:107] Waiting for socket to connect
0619 16:47:40.933028 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:44977: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":424,"thread_start_us":257,"threads_started":1}
I20260619 16:47:41.099874 17473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:47:41.099165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44977 (local address 127.0.97.1:40539)
0619 16:47:41.099622 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0619 16:47:41.099714 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:44977: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":355,"thread_start_us":248,"threads_started":1}
I20260619 16:47:41.519006 17474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:47:41.518269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44977 (local address 127.0.97.2:44007)
0619 16:47:41.518733 (+ 464us) negotiation.cc:107] Waiting for socket to connect
0619 16:47:41.518828 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:44977: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":360,"thread_start_us":228,"threads_started":1}
I20260619 16:47:41.934739 17475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:47:41.934045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44977 (local address 127.0.97.3:44679)
0619 16:47:41.934483 (+ 438us) negotiation.cc:107] Waiting for socket to connect
0619 16:47:41.934561 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:44977: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":333,"thread_start_us":218,"threads_started":1}
I20260619 16:47:42.101508 17476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:47:42.100762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44977 (local address 127.0.97.1:46995)
0619 16:47:42.101218 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0619 16:47:42.101312 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:44977: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":360,"thread_start_us":251,"threads_started":1}
I20260619 16:47:42.520773 17477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:47:42.519999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44977 (local address 127.0.97.2:54471)
0619 16:47:42.520514 (+ 515us) negotiation.cc:107] Waiting for socket to connect
0619 16:47:42.520601 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:44977: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":396,"thread_start_us":259,"threads_started":1}
I20260619 16:47:42.936431 17478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:47:42.935667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44977 (local address 127.0.97.3:53121)
0619 16:47:42.936176 (+ 509us) negotiation.cc:107] Waiting for socket to connect
0619 16:47:42.936263 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:44977: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":397,"thread_start_us":258,"threads_started":1}
I20260619 16:47:43.103147 17479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:47:43.102433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44977 (local address 127.0.97.1:33055)
0619 16:47:43.102903 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0619 16:47:43.102986 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:44977: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":362,"thread_start_us":214,"threads_started":1}
I20260619 16:47:43.522468 17480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:47:43.521794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44977 (local address 127.0.97.2:34673)
0619 16:47:43.522214 (+ 420us) negotiation.cc:107] Waiting for socket to connect
0619 16:47:43.522291 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:44977: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":325,"thread_start_us":202,"threads_started":1}
I20260619 16:47:43.938139 17481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:47:43.937330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44977 (local address 127.0.97.3:48123)
0619 16:47:43.937873 (+ 543us) negotiation.cc:107] Waiting for socket to connect
0619 16:47:43.937959 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:44977: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":370,"thread_start_us":239,"threads_started":1}
I20260619 16:47:44.104898 17482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:47:44.104076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44977 (local address 127.0.97.1:39059)
0619 16:47:44.104569 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0619 16:47:44.104660 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:44977: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":380,"thread_start_us":259,"threads_started":1}
I20260619 16:47:44.523998 17483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:47:44.523320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44977 (local address 127.0.97.2:35925)
0619 16:47:44.523756 (+ 436us) negotiation.cc:107] Waiting for socket to connect
0619 16:47:44.523841 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:44977: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":331,"thread_start_us":217,"threads_started":1}
I20260619 16:47:44.735635 17484 raft_consensus.cc:482] T b3e48dd34c004503af174b9e5958a3ef P be15a6b9c4824e229834519ec2b988d0 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 16:47:44.735845 17484 raft_consensus.cc:504] T b3e48dd34c004503af174b9e5958a3ef P be15a6b9c4824e229834519ec2b988d0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1efaebd5eb7a4bba9df40e4b0093546c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37569 } } peers { permanent_uuid: "be15a6b9c4824e229834519ec2b988d0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42315 } } peers { permanent_uuid: "8b323dc8ce0f412db10a79763cb324ce" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34415 } }
I20260619 16:47:44.736449 17484 leader_election.cc:288] T b3e48dd34c004503af174b9e5958a3ef P be15a6b9c4824e229834519ec2b988d0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1efaebd5eb7a4bba9df40e4b0093546c (127.0.97.3:37569), 8b323dc8ce0f412db10a79763cb324ce (127.0.97.2:34415)
W20260619 16:47:44.737433 17060 leader_election.cc:341] T b3e48dd34c004503af174b9e5958a3ef P be15a6b9c4824e229834519ec2b988d0 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1efaebd5eb7a4bba9df40e4b0093546c (127.0.97.3:37569): Not found: Tablet not found: b3e48dd34c004503af174b9e5958a3ef
W20260619 16:47:44.737684 17060 leader_election.cc:341] T b3e48dd34c004503af174b9e5958a3ef P be15a6b9c4824e229834519ec2b988d0 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 8b323dc8ce0f412db10a79763cb324ce (127.0.97.2:34415): Not found: Tablet not found: b3e48dd34c004503af174b9e5958a3ef
I20260619 16:47:44.737847 17060 leader_election.cc:302] T b3e48dd34c004503af174b9e5958a3ef P be15a6b9c4824e229834519ec2b988d0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: be15a6b9c4824e229834519ec2b988d0; no voters: 1efaebd5eb7a4bba9df40e4b0093546c, 8b323dc8ce0f412db10a79763cb324ce
I20260619 16:47:44.738099 17484 raft_consensus.cc:2726] T b3e48dd34c004503af174b9e5958a3ef P be15a6b9c4824e229834519ec2b988d0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 16:47:44.939720 17485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:47:44.939007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44977 (local address 127.0.97.3:57881)
0619 16:47:44.939478 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0619 16:47:44.939558 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:44977: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":357,"thread_start_us":231,"threads_started":1}
I20260619 16:47:45.106634 17486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:47:45.105826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44977 (local address 127.0.97.1:42555)
0619 16:47:45.106298 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0619 16:47:45.106392 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:44977: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":363,"thread_start_us":237,"threads_started":1}
I20260619 16:47:45.525599 17487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:47:45.524836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44977 (local address 127.0.97.2:50933)
0619 16:47:45.525310 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0619 16:47:45.525406 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:44977: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":361,"thread_start_us":235,"threads_started":1}
I20260619 16:47:45.941306 17488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:47:45.940580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44977 (local address 127.0.97.3:57663)
0619 16:47:45.941011 (+ 431us) negotiation.cc:107] Waiting for socket to connect
0619 16:47:45.941093 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:44977: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":336,"thread_start_us":235,"threads_started":1}
I20260619 16:47:46.108577 17489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:47:46.107860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44977 (local address 127.0.97.1:52179)
0619 16:47:46.108302 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0619 16:47:46.108398 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:44977: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":334,"thread_start_us":222,"threads_started":1}
I20260619 16:47:46.527375 17490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:47:46.526715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44977 (local address 127.0.97.2:37145)
0619 16:47:46.527133 (+ 418us) negotiation.cc:107] Waiting for socket to connect
0619 16:47:46.527224 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:44977: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":316,"thread_start_us":203,"threads_started":1}
I20260619 16:47:46.923161 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu pid 17002
I20260619 16:47:46.937310 17491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:47:46.924238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51070 (local address 127.0.97.1:42315)
0619 16:47:46.924800 (+ 562us) server_negotiation.cc:206] Beginning negotiation
0619 16:47:46.924807 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:47:46.924830 (+ 23us) server_negotiation.cc:407] Connection header received
0619 16:47:46.924887 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:47:46.924893 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:47:46.925098 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:47:46.925242 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:47:46.926639 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:47:46.927576 (+ 937us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:47:46.928861 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:47:46.929440 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:47:46.929567 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:47:46.931962 (+ 2395us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:47:46.931973 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:47:46.931987 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:47:46.932021 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:47:46.934683 (+ 2662us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:47:46.935284 (+ 601us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:47:46.935291 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:47:46.935293 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:47:46.935441 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:47:46.935782 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:47:46.935787 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:47:46.935789 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:47:46.936125 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:47:46.936402 (+ 277us) server_negotiation.cc:1035] Waiting for connection context
0619 16:47:46.936731 (+ 329us) server_negotiation.cc:299] Negotiation successful
0619 16:47:46.936992 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":375,"thread_start_us":240,"threads_started":1}
I20260619 16:47:46.938541 17167 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:47:46.943199 17493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:47:46.942423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44977 (local address 127.0.97.3:51481)
0619 16:47:46.942903 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0619 16:47:46.943016 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:44977: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":372,"thread_start_us":253,"threads_started":1}
I20260619 16:47:47.027583 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu with pid 17002
I20260619 16:47:47.042178 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu pid 17175
I20260619 16:47:47.055464 17494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:47:47.043008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51652 (local address 127.0.97.2:34415)
0619 16:47:47.043428 (+ 420us) server_negotiation.cc:206] Beginning negotiation
0619 16:47:47.043433 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:47:47.043448 (+ 15us) server_negotiation.cc:407] Connection header received
0619 16:47:47.043503 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:47:47.043506 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:47:47.043565 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:47:47.043658 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:47:47.044998 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:47:47.045826 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:47:47.047081 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:47:47.047507 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:47:47.047602 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:47:47.050139 (+ 2537us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:47:47.050144 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:47:47.050148 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:47:47.050182 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:47:47.052975 (+ 2793us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:47:47.053699 (+ 724us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:47:47.053703 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:47:47.053706 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:47:47.053902 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:47:47.054276 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:47:47.054279 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:47:47.054281 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:47:47.054550 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:47:47.054774 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0619 16:47:47.054970 (+ 196us) server_negotiation.cc:299] Negotiation successful
0619 16:47:47.055141 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":330,"thread_start_us":229,"threads_started":1}
I20260619 16:47:47.056107 17299 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:47:47.139999 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu with pid 17175
I20260619 16:47:47.154193 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu pid 17307
I20260619 16:47:47.166939 17496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:47:47.155040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38300 (local address 127.0.97.3:37569)
0619 16:47:47.155605 (+ 565us) server_negotiation.cc:206] Beginning negotiation
0619 16:47:47.155610 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:47:47.155627 (+ 17us) server_negotiation.cc:407] Connection header received
0619 16:47:47.155684 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:47:47.155688 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:47:47.155757 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:47:47.155850 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:47:47.157192 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:47:47.158026 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:47:47.159250 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:47:47.159639 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:47:47.159730 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:47:47.162139 (+ 2409us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:47:47.162144 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:47:47.162147 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:47:47.162177 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:47:47.164775 (+ 2598us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:47:47.165375 (+ 600us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:47:47.165379 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:47:47.165381 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:47:47.165505 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:47:47.165835 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:47:47.165838 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:47:47.165840 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:47:47.166081 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:47:47.166279 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0619 16:47:47.166500 (+ 221us) server_negotiation.cc:299] Negotiation successful
0619 16:47:47.166656 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":452,"thread_start_us":304,"threads_started":1}
I20260619 16:47:47.167563 17430 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:47:47.252013 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu with pid 17307
2026-06-19T16:47:47Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (63776 ms)
[----------] 30 tests from SecurityITest (281311 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887385903558-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 16:47:47 dist-test-slave-wvz7 krb5kdc[17500](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 16:47:47 dist-test-slave-wvz7 krb5kdc[17500](info): set up 2 sockets
Jun 19 16:47:47 dist-test-slave-wvz7 krb5kdc[17500](info): commencing operation
krb5kdc: starting...
W20260619 16:47:48.265525 388 mini_kdc.cc:121] Time spent starting KDC: real 0.977s user 0.006s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887385903558-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887385903558-388-0/krb5kdc/test-user.keytab.
Jun 19 16:47:48 dist-test-slave-wvz7 krb5kdc[17500](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887668, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T16:47:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T16:47:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887385903558-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887385903558-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260619 16:47:48.487365 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu
/tmp/dist-test-taskEs2EH8/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887385903558-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887385903558-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887385903558-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887385903558-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:36589
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887385903558-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45637
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:36589
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887385903558-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887385903558-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887385903558-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:47:48.797550 17513 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:47:48.797838 17513 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:47:48.797933 17513 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:47:48.804883 17513 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:47:48.804979 17513 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:47:48.805023 17513 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:47:48.805078 17513 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:47:48.805120 17513 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:47:48.814585 17513 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45637
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887385903558-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887385903558-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:36589
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887385903558-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:36589
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887385903558-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887385903558-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:47:48.815793 17513 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:47:48.817037 17513 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:47:48.827841 17521 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:47:48.827898 17518 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:47:48.827925 17519 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:47:48.828728 17513 server_base.cc:1034] running on GCE node
I20260619 16:47:48.829465 17513 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:47:48.830950 17513 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:47:48.832151 17513 hybrid_clock.cc:648] HybridClock initialized: now 1781887668832102 us; error 48 us; skew 500 ppm
I20260619 16:47:48.832465 17513 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:47:48 dist-test-slave-wvz7 krb5kdc[17500](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887668, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:47:48.835381 17513 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260619 16:47:48.836761 17513 webserver.cc:460] Webserver started at http://127.0.97.62:46137/ using document root <none> and password file <none>
I20260619 16:47:48.837441 17513 fs_manager.cc:359] Metadata directory not provided
I20260619 16:47:48.837556 17513 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:47:48.837872 17513 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:47:48.840611 17513 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887385903558-388-0/minicluster-data/master-0/data/instance:
uuid: "258a86eabe5e42368438b37b4c4936fa"
format_stamp: "Formatted at 2026-06-19 16:47:48 on dist-test-slave-wvz7"
server_key: "5be453821054a020f1e30cae76592c6c"
server_key_iv: "efd524303c13d9b772425ac3428edfd0"
server_key_version: "encryptionkey@0"
I20260619 16:47:48.841289 17513 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887385903558-388-0/minicluster-data/master-0/wal/instance:
uuid: "258a86eabe5e42368438b37b4c4936fa"
format_stamp: "Formatted at 2026-06-19 16:47:48 on dist-test-slave-wvz7"
server_key: "5be453821054a020f1e30cae76592c6c"
server_key_iv: "efd524303c13d9b772425ac3428edfd0"
server_key_version: "encryptionkey@0"
I20260619 16:47:48.846130 17513 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260619 16:47:48.849287 17528 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:47:48.850509 17513 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 16:47:48.850665 17513 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887385903558-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887385903558-388-0/minicluster-data/master-0/wal
uuid: "258a86eabe5e42368438b37b4c4936fa"
format_stamp: "Formatted at 2026-06-19 16:47:48 on dist-test-slave-wvz7"
server_key: "5be453821054a020f1e30cae76592c6c"
server_key_iv: "efd524303c13d9b772425ac3428edfd0"
server_key_version: "encryptionkey@0"
I20260619 16:47:48.850829 17513 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887385903558-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887385903558-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887385903558-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:47:48.868963 17513 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:47:48.873108 17513 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:47:48.873389 17513 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:47:48.890795 17513 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:36589
I20260619 16:47:48.890854 17579 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:36589 every 8 connection(s)
I20260619 16:47:48.892136 17513 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887385903558-388-0/minicluster-data/master-0/data/info.pb
I20260619 16:47:48.896008 17580 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:47:48.898284 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu as pid 17513
I20260619 16:47:48.898463 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887385903558-388-0/minicluster-data/master-0/wal/instance
I20260619 16:47:48.899104 388 external_mini_cluster.cc:1428] Setting key 71ce79a83a7e8a0adbc926845c730646
I20260619 16:47:48.907774 17580 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 258a86eabe5e42368438b37b4c4936fa: Bootstrap starting.
I20260619 16:47:48.911401 17580 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 258a86eabe5e42368438b37b4c4936fa: Neither blocks nor log segments found. Creating new log.
I20260619 16:47:48.912662 17580 log.cc:824] T 00000000000000000000000000000000 P 258a86eabe5e42368438b37b4c4936fa: Log is configured to *not* fsync() on all Append() calls
Jun 19 16:47:48 dist-test-slave-wvz7 krb5kdc[17500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887668, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 16:47:48.915571 17580 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 258a86eabe5e42368438b37b4c4936fa: No bootstrap required, opened a new log
I20260619 16:47:48.922005 17583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:47:48.901007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39382 (local address 127.0.97.62:36589)
0619 16:47:48.901794 (+ 787us) server_negotiation.cc:206] Beginning negotiation
0619 16:47:48.901808 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 16:47:48.901851 (+ 43us) server_negotiation.cc:407] Connection header received
0619 16:47:48.903483 (+ 1632us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:47:48.903505 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:47:48.903878 (+ 373us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:47:48.904202 (+ 324us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:47:48.906068 (+ 1866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:47:48.907389 (+ 1321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:47:48.909172 (+ 1783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:47:48.909979 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:47:48.910141 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:47:48.914913 (+ 4772us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:47:48.914920 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:47:48.914935 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:47:48.914970 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:47:48.918019 (+ 3049us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:47:48.918670 (+ 651us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:47:48.918677 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:47:48.918684 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:47:48.918848 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:47:48.919241 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:47:48.919245 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:47:48.919247 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:47:48.919797 (+ 550us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:47:48.920080 (+ 283us) server_negotiation.cc:1035] Waiting for connection context
0619 16:47:48.920519 (+ 439us) server_negotiation.cc:299] Negotiation successful
0619 16:47:48.920939 (+ 420us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":555,"thread_start_us":339,"threads_started":1}
I20260619 16:47:48.923336 17580 raft_consensus.cc:348] T 00000000000000000000000000000000 P 258a86eabe5e42368438b37b4c4936fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "258a86eabe5e42368438b37b4c4936fa" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36589 } }
I20260619 16:47:48.923681 17580 raft_consensus.cc:374] T 00000000000000000000000000000000 P 258a86eabe5e42368438b37b4c4936fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:47:48.923776 17580 raft_consensus.cc:729] T 00000000000000000000000000000000 P 258a86eabe5e42368438b37b4c4936fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 258a86eabe5e42368438b37b4c4936fa, State: Initialized, Role: FOLLOWER
I20260619 16:47:48.924296 17580 consensus_queue.cc:260] T 00000000000000000000000000000000 P 258a86eabe5e42368438b37b4c4936fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "258a86eabe5e42368438b37b4c4936fa" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36589 } }
I20260619 16:47:48.924535 17580 raft_consensus.cc:388] T 00000000000000000000000000000000 P 258a86eabe5e42368438b37b4c4936fa [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:47:48.924698 17580 raft_consensus.cc:482] T 00000000000000000000000000000000 P 258a86eabe5e42368438b37b4c4936fa [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:47:48.924865 17580 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 258a86eabe5e42368438b37b4c4936fa [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:47:48.927016 17580 raft_consensus.cc:504] T 00000000000000000000000000000000 P 258a86eabe5e42368438b37b4c4936fa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "258a86eabe5e42368438b37b4c4936fa" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36589 } }
I20260619 16:47:48.927454 17580 leader_election.cc:302] T 00000000000000000000000000000000 P 258a86eabe5e42368438b37b4c4936fa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 258a86eabe5e42368438b37b4c4936fa; no voters:
I20260619 16:47:48.927980 17580 leader_election.cc:288] T 00000000000000000000000000000000 P 258a86eabe5e42368438b37b4c4936fa [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 16:47:48.928134 17585 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 258a86eabe5e42368438b37b4c4936fa [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:47:48.929224 17585 raft_consensus.cc:686] T 00000000000000000000000000000000 P 258a86eabe5e42368438b37b4c4936fa [term 1 LEADER]: Becoming Leader. State: Replica: 258a86eabe5e42368438b37b4c4936fa, State: Running, Role: LEADER
I20260619 16:47:48.929908 17585 consensus_queue.cc:237] T 00000000000000000000000000000000 P 258a86eabe5e42368438b37b4c4936fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "258a86eabe5e42368438b37b4c4936fa" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36589 } }
I20260619 16:47:48.930238 17580 sys_catalog.cc:564] T 00000000000000000000000000000000 P 258a86eabe5e42368438b37b4c4936fa [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:47:48.932593 17587 sys_catalog.cc:455] T 00000000000000000000000000000000 P 258a86eabe5e42368438b37b4c4936fa [sys.catalog]: SysCatalogTable state changed. Reason: New leader 258a86eabe5e42368438b37b4c4936fa. Latest consensus state: current_term: 1 leader_uuid: "258a86eabe5e42368438b37b4c4936fa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "258a86eabe5e42368438b37b4c4936fa" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36589 } } }
I20260619 16:47:48.932987 17587 sys_catalog.cc:458] T 00000000000000000000000000000000 P 258a86eabe5e42368438b37b4c4936fa [sys.catalog]: This master's current role is: LEADER
I20260619 16:47:48.933506 17586 sys_catalog.cc:455] T 00000000000000000000000000000000 P 258a86eabe5e42368438b37b4c4936fa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "258a86eabe5e42368438b37b4c4936fa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "258a86eabe5e42368438b37b4c4936fa" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36589 } } }
I20260619 16:47:48.933806 17586 sys_catalog.cc:458] T 00000000000000000000000000000000 P 258a86eabe5e42368438b37b4c4936fa [sys.catalog]: This master's current role is: LEADER
I20260619 16:47:48.939000 17592 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:47:48.944443 17592 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:47:48.953627 17592 catalog_manager.cc:1347] Generated new cluster ID: ddbc7b6a9aa94cf39acec3aa3104f675
I20260619 16:47:48.953786 17592 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:47:49.004897 17592 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 16:47:49.006315 17592 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:47:49.017361 17592 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 258a86eabe5e42368438b37b4c4936fa: Generated new TSK 0
I20260619 16:47:49.018263 17592 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887385903558-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887385903558-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260619 16:47:49.081084 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu
/tmp/dist-test-taskEs2EH8/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887385903558-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887385903558-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887385903558-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887385903558-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36589
--keytab_file=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887385903558-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45637
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887385903558-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887385903558-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887385903558-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:47:49.403254 17606 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:47:49.403556 17606 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:47:49.403697 17606 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:47:49.411008 17606 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:47:49.411144 17606 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:47:49.411299 17606 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260619 16:47:49.421262 17606 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45637
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887385903558-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887385903558-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887385903558-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887385903558-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36589
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887385903558-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:47:49.422595 17606 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:47:49.424010 17606 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:47:49.436764 17611 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:47:49.436817 17614 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:47:49.436795 17612 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:47:49.437650 17606 server_base.cc:1034] running on GCE node
I20260619 16:47:49.438347 17606 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:47:49.439388 17606 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:47:49.440659 17606 hybrid_clock.cc:648] HybridClock initialized: now 1781887669440592 us; error 54 us; skew 500 ppm
I20260619 16:47:49.440965 17606 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:47:49 dist-test-slave-wvz7 krb5kdc[17500](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887669, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:47:49.443964 17606 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260619 16:47:49.445426 17606 webserver.cc:460] Webserver started at http://127.0.97.1:37991/ using document root <none> and password file <none>
I20260619 16:47:49.446139 17606 fs_manager.cc:359] Metadata directory not provided
I20260619 16:47:49.446251 17606 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:47:49.446525 17606 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:47:49.449294 17606 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887385903558-388-0/minicluster-data/ts-0/data/instance:
uuid: "a357c0f5f58d4bd1930248219ca97caf"
format_stamp: "Formatted at 2026-06-19 16:47:49 on dist-test-slave-wvz7"
server_key: "c1bec06455303534cf7c337a82817ae3"
server_key_iv: "ab7a5d6a70fb320a4ed3da2f976e3ea3"
server_key_version: "encryptionkey@0"
I20260619 16:47:49.450074 17606 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887385903558-388-0/minicluster-data/ts-0/wal/instance:
uuid: "a357c0f5f58d4bd1930248219ca97caf"
format_stamp: "Formatted at 2026-06-19 16:47:49 on dist-test-slave-wvz7"
server_key: "c1bec06455303534cf7c337a82817ae3"
server_key_iv: "ab7a5d6a70fb320a4ed3da2f976e3ea3"
server_key_version: "encryptionkey@0"
I20260619 16:47:49.455008 17606 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260619 16:47:49.458374 17621 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:47:49.459645 17606 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 16:47:49.459789 17606 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887385903558-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887385903558-388-0/minicluster-data/ts-0/wal
uuid: "a357c0f5f58d4bd1930248219ca97caf"
format_stamp: "Formatted at 2026-06-19 16:47:49 on dist-test-slave-wvz7"
server_key: "c1bec06455303534cf7c337a82817ae3"
server_key_iv: "ab7a5d6a70fb320a4ed3da2f976e3ea3"
server_key_version: "encryptionkey@0"
I20260619 16:47:49.459959 17606 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887385903558-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887385903558-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887385903558-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:47:49.473129 17606 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:47:49.476951 17606 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:47:49.477214 17606 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:47:49.478221 17606 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:47:49.479494 17606 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:47:49.479568 17606 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:47:49.479698 17606 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:47:49.479755 17606 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:47:49.509187 17606 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:35917
I20260619 16:47:49.509253 17733 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:35917 every 8 connection(s)
I20260619 16:47:49.510448 17606 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887385903558-388-0/minicluster-data/ts-0/data/info.pb
I20260619 16:47:49.516279 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu as pid 17606
I20260619 16:47:49.516475 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887385903558-388-0/minicluster-data/ts-0/wal/instance
I20260619 16:47:49.517309 388 external_mini_cluster.cc:1428] Setting key eb94ea4e7f1a1f1ee5561950a8ab50c9
Jun 19 16:47:49 dist-test-slave-wvz7 krb5kdc[17500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887669, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 16:47:49.531464 17736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:47:49.512929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:56015 (local address 127.0.97.62:36589)
0619 16:47:49.513333 (+ 404us) server_negotiation.cc:206] Beginning negotiation
0619 16:47:49.513337 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:47:49.514313 (+ 976us) server_negotiation.cc:407] Connection header received
0619 16:47:49.516296 (+ 1983us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:47:49.516303 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:47:49.516392 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:47:49.516572 (+ 180us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:47:49.518541 (+ 1969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:47:49.519584 (+ 1043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:47:49.521794 (+ 2210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:47:49.522207 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:47:49.522304 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:47:49.526028 (+ 3724us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:47:49.526033 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:47:49.526036 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:47:49.526068 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:47:49.528464 (+ 2396us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:47:49.529155 (+ 691us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:47:49.529159 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:47:49.529161 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:47:49.529276 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:47:49.529687 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:47:49.529690 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:47:49.529693 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:47:49.529999 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:47:49.530145 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0619 16:47:49.530876 (+ 731us) server_negotiation.cc:299] Negotiation successful
0619 16:47:49.531106 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":302,"thread_start_us":204,"threads_started":1}
I20260619 16:47:49.532572 17737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:47:49.513400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36589 (local address 127.0.97.1:56015)
0619 16:47:49.514153 (+ 753us) negotiation.cc:107] Waiting for socket to connect
0619 16:47:49.514195 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0619 16:47:49.515930 (+ 1735us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:47:49.516808 (+ 878us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:47:49.516820 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:47:49.517579 (+ 759us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:47:49.518365 (+ 786us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:47:49.518390 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:47:49.519726 (+ 1336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:47:49.519731 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:47:49.521615 (+ 1884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:47:49.521631 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:47:49.522415 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:47:49.522421 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:47:49.522690 (+ 269us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:47:49.523349 (+ 659us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:47:49.523374 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:47:49.525821 (+ 2447us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:47:49.528602 (+ 2781us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:47:49.528611 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:47:49.528628 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:47:49.529034 (+ 406us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:47:49.529376 (+ 342us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:47:49.529381 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:47:49.529385 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:47:49.529570 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:47:49.530129 (+ 559us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:47:49.530145 (+ 16us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:47:49.530478 (+ 333us) client_negotiation.cc:769] Sending connection context
0619 16:47:49.530744 (+ 266us) client_negotiation.cc:240] Negotiation successful
0619 16:47:49.531351 (+ 607us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":500,"thread_start_us":318,"threads_started":1}
I20260619 16:47:49.534150 17734 heartbeater.cc:347] Connected to a master server at 127.0.97.62:36589
I20260619 16:47:49.534477 17734 heartbeater.cc:464] Registering TS with master...
I20260619 16:47:49.535346 17734 heartbeater.cc:511] Master 127.0.97.62:36589 requested a full tablet report, sending...
I20260619 16:47:49.537909 17545 ts_manager.cc:194] Registered new tserver with Master: a357c0f5f58d4bd1930248219ca97caf (127.0.97.1:35917)
I20260619 16:47:49.539991 17545 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:56015
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887385903558-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887385903558-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260619 16:47:49.568567 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu
/tmp/dist-test-taskEs2EH8/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887385903558-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887385903558-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887385903558-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887385903558-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36589
--keytab_file=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887385903558-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45637
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887385903558-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887385903558-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887385903558-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:47:49.882656 17740 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:47:49.882923 17740 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:47:49.883056 17740 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:47:49.892845 17740 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:47:49.892972 17740 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:47:49.893154 17740 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260619 16:47:49.908180 17740 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45637
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887385903558-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887385903558-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887385903558-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887385903558-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36589
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887385903558-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:47:49.909698 17740 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:47:49.911268 17740 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:47:49.924027 17745 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:47:49.923995 17746 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:47:49.923975 17748 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:47:49.924832 17740 server_base.cc:1034] running on GCE node
I20260619 16:47:49.925484 17740 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:47:49.926564 17740 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:47:49.927783 17740 hybrid_clock.cc:648] HybridClock initialized: now 1781887669927710 us; error 60 us; skew 500 ppm
I20260619 16:47:49.928112 17740 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:47:49 dist-test-slave-wvz7 krb5kdc[17500](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887669, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:47:49.931458 17740 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260619 16:47:49.933018 17740 webserver.cc:460] Webserver started at http://127.0.97.2:43519/ using document root <none> and password file <none>
I20260619 16:47:49.933683 17740 fs_manager.cc:359] Metadata directory not provided
I20260619 16:47:49.933844 17740 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:47:49.934134 17740 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:47:49.937103 17740 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887385903558-388-0/minicluster-data/ts-1/data/instance:
uuid: "ad563589f19344e49bc21900142c3c27"
format_stamp: "Formatted at 2026-06-19 16:47:49 on dist-test-slave-wvz7"
server_key: "c8e96849226b337fc373cf5a675457f5"
server_key_iv: "2893625bc20d0ef8694f32fac7bd1445"
server_key_version: "encryptionkey@0"
I20260619 16:47:49.937955 17740 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887385903558-388-0/minicluster-data/ts-1/wal/instance:
uuid: "ad563589f19344e49bc21900142c3c27"
format_stamp: "Formatted at 2026-06-19 16:47:49 on dist-test-slave-wvz7"
server_key: "c8e96849226b337fc373cf5a675457f5"
server_key_iv: "2893625bc20d0ef8694f32fac7bd1445"
server_key_version: "encryptionkey@0"
I20260619 16:47:49.943329 17740 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260619 16:47:49.947237 17755 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:47:49.948616 17740 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 16:47:49.948755 17740 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887385903558-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887385903558-388-0/minicluster-data/ts-1/wal
uuid: "ad563589f19344e49bc21900142c3c27"
format_stamp: "Formatted at 2026-06-19 16:47:49 on dist-test-slave-wvz7"
server_key: "c8e96849226b337fc373cf5a675457f5"
server_key_iv: "2893625bc20d0ef8694f32fac7bd1445"
server_key_version: "encryptionkey@0"
I20260619 16:47:49.948925 17740 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887385903558-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887385903558-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887385903558-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:47:49.971169 17740 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:47:49.974992 17740 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:47:49.975227 17740 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:47:49.976133 17740 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:47:49.977349 17740 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:47:49.977422 17740 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:47:49.977509 17740 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:47:49.977548 17740 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:47:50.007901 17740 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:44633
I20260619 16:47:50.007958 17867 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:44633 every 8 connection(s)
I20260619 16:47:50.009209 17740 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887385903558-388-0/minicluster-data/ts-1/data/info.pb
I20260619 16:47:50.010581 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu as pid 17740
I20260619 16:47:50.010757 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887385903558-388-0/minicluster-data/ts-1/wal/instance
I20260619 16:47:50.011325 388 external_mini_cluster.cc:1428] Setting key e2c3426308411955e959e5704d7e7ddf
Jun 19 16:47:50 dist-test-slave-wvz7 krb5kdc[17500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887669, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 16:47:50.030800 17736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:47:50.012056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:33731 (local address 127.0.97.62:36589)
0619 16:47:50.012275 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0619 16:47:50.012279 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:47:50.013533 (+ 1254us) server_negotiation.cc:407] Connection header received
0619 16:47:50.015468 (+ 1935us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:47:50.015474 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:47:50.015570 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:47:50.015657 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:47:50.017275 (+ 1618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:47:50.018113 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:47:50.019692 (+ 1579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:47:50.020253 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:47:50.020387 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:47:50.024374 (+ 3987us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:47:50.024381 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:47:50.024385 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:47:50.024429 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:47:50.027581 (+ 3152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:47:50.028396 (+ 815us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:47:50.028401 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:47:50.028404 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:47:50.028547 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:47:50.028944 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:47:50.028949 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:47:50.028952 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:47:50.029279 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:47:50.029416 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0619 16:47:50.030196 (+ 780us) server_negotiation.cc:299] Negotiation successful
0619 16:47:50.030427 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260619 16:47:50.031780 17870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:47:50.012549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36589 (local address 127.0.97.2:33731)
0619 16:47:50.013376 (+ 827us) negotiation.cc:107] Waiting for socket to connect
0619 16:47:50.013420 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0619 16:47:50.015175 (+ 1755us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:47:50.015927 (+ 752us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:47:50.015936 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:47:50.016502 (+ 566us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:47:50.017094 (+ 592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:47:50.017113 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:47:50.018286 (+ 1173us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:47:50.018290 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:47:50.019545 (+ 1255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:47:50.019557 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:47:50.020499 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:47:50.020504 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:47:50.020762 (+ 258us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:47:50.021465 (+ 703us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:47:50.021488 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:47:50.024068 (+ 2580us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:47:50.027753 (+ 3685us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:47:50.027761 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:47:50.027776 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:47:50.028280 (+ 504us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:47:50.028656 (+ 376us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:47:50.028659 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:47:50.028662 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:47:50.028849 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:47:50.029397 (+ 548us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:47:50.029404 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:47:50.029841 (+ 437us) client_negotiation.cc:769] Sending connection context
0619 16:47:50.030095 (+ 254us) client_negotiation.cc:240] Negotiation successful
0619 16:47:50.030552 (+ 457us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":517,"thread_start_us":299,"threads_started":1}
I20260619 16:47:50.033311 17868 heartbeater.cc:347] Connected to a master server at 127.0.97.62:36589
I20260619 16:47:50.033646 17868 heartbeater.cc:464] Registering TS with master...
I20260619 16:47:50.034538 17868 heartbeater.cc:511] Master 127.0.97.62:36589 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260619 16:47:50.036090 17545 ts_manager.cc:194] Registered new tserver with Master: ad563589f19344e49bc21900142c3c27 (127.0.97.2:44633)
I20260619 16:47:50.036967 17545 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:33731
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887385903558-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887385903558-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260619 16:47:50.063086 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu
/tmp/dist-test-taskEs2EH8/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887385903558-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887385903558-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887385903558-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887385903558-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36589
--keytab_file=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887385903558-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45637
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887385903558-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887385903558-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887385903558-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:47:50.379891 17873 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:47:50.380179 17873 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:47:50.380312 17873 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:47:50.387765 17873 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:47:50.387880 17873 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:47:50.388033 17873 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260619 16:47:50.397722 17873 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45637
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887385903558-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887385903558-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887385903558-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887385903558-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36589
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887385903558-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:47:50.398998 17873 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:47:50.400242 17873 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:47:50.412696 17878 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:47:50.412791 17879 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:47:50.412734 17881 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:47:50.413452 17873 server_base.cc:1034] running on GCE node
I20260619 16:47:50.414041 17873 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:47:50.414923 17873 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:47:50.416108 17873 hybrid_clock.cc:648] HybridClock initialized: now 1781887670416063 us; error 45 us; skew 500 ppm
I20260619 16:47:50.416397 17873 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:47:50 dist-test-slave-wvz7 krb5kdc[17500](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887670, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:47:50.424126 17873 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260619 16:47:50.425637 17873 webserver.cc:460] Webserver started at http://127.0.97.3:45797/ using document root <none> and password file <none>
I20260619 16:47:50.426359 17873 fs_manager.cc:359] Metadata directory not provided
I20260619 16:47:50.426476 17873 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:47:50.426759 17873 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:47:50.429541 17873 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887385903558-388-0/minicluster-data/ts-2/data/instance:
uuid: "125454bf52364613932bbe122513f66e"
format_stamp: "Formatted at 2026-06-19 16:47:50 on dist-test-slave-wvz7"
server_key: "67658e17ac693edc995851a699a3da8d"
server_key_iv: "3ac45b24926052e007ebada867c719dd"
server_key_version: "encryptionkey@0"
I20260619 16:47:50.430321 17873 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887385903558-388-0/minicluster-data/ts-2/wal/instance:
uuid: "125454bf52364613932bbe122513f66e"
format_stamp: "Formatted at 2026-06-19 16:47:50 on dist-test-slave-wvz7"
server_key: "67658e17ac693edc995851a699a3da8d"
server_key_iv: "3ac45b24926052e007ebada867c719dd"
server_key_version: "encryptionkey@0"
I20260619 16:47:50.435181 17873 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260619 16:47:50.438663 17888 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:47:50.440088 17873 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 16:47:50.440260 17873 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887385903558-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887385903558-388-0/minicluster-data/ts-2/wal
uuid: "125454bf52364613932bbe122513f66e"
format_stamp: "Formatted at 2026-06-19 16:47:50 on dist-test-slave-wvz7"
server_key: "67658e17ac693edc995851a699a3da8d"
server_key_iv: "3ac45b24926052e007ebada867c719dd"
server_key_version: "encryptionkey@0"
I20260619 16:47:50.440444 17873 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887385903558-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887385903558-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887385903558-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:47:50.464011 17873 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:47:50.468176 17873 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:47:50.468461 17873 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:47:50.469455 17873 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:47:50.470825 17873 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:47:50.470898 17873 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:47:50.471009 17873 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:47:50.471058 17873 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:47:50.500639 17873 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:42965
I20260619 16:47:50.500675 18000 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:42965 every 8 connection(s)
I20260619 16:47:50.501845 17873 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887385903558-388-0/minicluster-data/ts-2/data/info.pb
I20260619 16:47:50.504940 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu as pid 17873
I20260619 16:47:50.505093 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887385903558-388-0/minicluster-data/ts-2/wal/instance
I20260619 16:47:50.505497 388 external_mini_cluster.cc:1428] Setting key 4d4fa43d864314f6b3727b8cb389f0a7
Jun 19 16:47:50 dist-test-slave-wvz7 krb5kdc[17500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887670, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 16:47:50.521759 17736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:47:50.504201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:39487 (local address 127.0.97.62:36589)
0619 16:47:50.504369 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0619 16:47:50.504373 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:47:50.505609 (+ 1236us) server_negotiation.cc:407] Connection header received
0619 16:47:50.507878 (+ 2269us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:47:50.507882 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:47:50.507946 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:47:50.508039 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:47:50.509683 (+ 1644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:47:50.510843 (+ 1160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:47:50.512387 (+ 1544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:47:50.512975 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:47:50.513099 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:47:50.516434 (+ 3335us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:47:50.516439 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:47:50.516442 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:47:50.516484 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:47:50.518803 (+ 2319us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:47:50.519535 (+ 732us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:47:50.519541 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:47:50.519543 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:47:50.519719 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:47:50.520105 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:47:50.520111 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:47:50.520114 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:47:50.520390 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:47:50.520523 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0619 16:47:50.521185 (+ 662us) server_negotiation.cc:299] Negotiation successful
0619 16:47:50.521414 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260619 16:47:50.522656 18003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:47:50.504669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36589 (local address 127.0.97.3:39487)
0619 16:47:50.505450 (+ 781us) negotiation.cc:107] Waiting for socket to connect
0619 16:47:50.505497 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0619 16:47:50.507619 (+ 2122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:47:50.508294 (+ 675us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:47:50.508305 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:47:50.508931 (+ 626us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:47:50.509490 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:47:50.509507 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:47:50.510969 (+ 1462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:47:50.510973 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:47:50.512220 (+ 1247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:47:50.512232 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:47:50.513174 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:47:50.513179 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:47:50.513384 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:47:50.514078 (+ 694us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:47:50.514102 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:47:50.516232 (+ 2130us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:47:50.518928 (+ 2696us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:47:50.518937 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:47:50.518952 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:47:50.519387 (+ 435us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:47:50.519816 (+ 429us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:47:50.519821 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:47:50.519824 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:47:50.520004 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:47:50.520519 (+ 515us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:47:50.520527 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:47:50.520833 (+ 306us) client_negotiation.cc:769] Sending connection context
0619 16:47:50.521062 (+ 229us) client_negotiation.cc:240] Negotiation successful
0619 16:47:50.521566 (+ 504us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":521,"thread_start_us":325,"threads_started":1}
I20260619 16:47:50.523864 18001 heartbeater.cc:347] Connected to a master server at 127.0.97.62:36589
I20260619 16:47:50.524111 18001 heartbeater.cc:464] Registering TS with master...
I20260619 16:47:50.524729 18001 heartbeater.cc:511] Master 127.0.97.62:36589 requested a full tablet report, sending...
I20260619 16:47:50.526026 17545 ts_manager.cc:194] Registered new tserver with Master: 125454bf52364613932bbe122513f66e (127.0.97.3:42965)
I20260619 16:47:50.526815 17545 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:39487
I20260619 16:47:50.532789 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 16:47:50.543354 17734 heartbeater.cc:503] Master 127.0.97.62:36589 was elected leader, sending a full tablet report...
I20260619 16:47:50.549116 17736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:47:50.537063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39394 (local address 127.0.97.62:36589)
0619 16:47:50.537275 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0619 16:47:50.537281 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:47:50.537573 (+ 292us) server_negotiation.cc:407] Connection header received
0619 16:47:50.537700 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:47:50.537703 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:47:50.537797 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:47:50.537871 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:47:50.539198 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:47:50.539974 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:47:50.541260 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:47:50.541662 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:47:50.541794 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:47:50.544363 (+ 2569us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:47:50.544368 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:47:50.544371 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:47:50.544401 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:47:50.546957 (+ 2556us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:47:50.547605 (+ 648us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:47:50.547610 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:47:50.547612 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:47:50.547723 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:47:50.547981 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:47:50.547984 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:47:50.547986 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:47:50.548280 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:47:50.548451 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0619 16:47:50.548690 (+ 239us) server_negotiation.cc:299] Negotiation successful
0619 16:47:50.548840 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260619 16:47:50.552898 17545 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39394:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 16:47:50.559726 17545 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 16:47:50.582542 18016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:47:50.572352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44108 (local address 127.0.97.2:44633)
0619 16:47:50.572975 (+ 623us) server_negotiation.cc:206] Beginning negotiation
0619 16:47:50.572985 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 16:47:50.574454 (+ 1469us) server_negotiation.cc:407] Connection header received
0619 16:47:50.574907 (+ 453us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:47:50.574916 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:47:50.575151 (+ 235us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:47:50.575327 (+ 176us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:47:50.576680 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:47:50.578193 (+ 1513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:47:50.580168 (+ 1975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:47:50.581291 (+ 1123us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:47:50.581421 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:47:50.581666 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0619 16:47:50.581987 (+ 321us) server_negotiation.cc:299] Negotiation successful
0619 16:47:50.582164 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":454,"thread_start_us":340,"threads_started":1}
I20260619 16:47:50.582638 18014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:47:50.571827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:44633 (local address 127.0.0.1:44108)
0619 16:47:50.574352 (+ 2525us) negotiation.cc:107] Waiting for socket to connect
0619 16:47:50.574369 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 16:47:50.574712 (+ 343us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:47:50.575614 (+ 902us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:47:50.575617 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:47:50.575649 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:47:50.576498 (+ 849us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:47:50.576510 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:47:50.578337 (+ 1827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:47:50.578345 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:47:50.580018 (+ 1673us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:47:50.580030 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:47:50.581498 (+ 1468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:47:50.581504 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:47:50.581722 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:47:50.581807 (+ 85us) client_negotiation.cc:769] Sending connection context
0619 16:47:50.581956 (+ 149us) client_negotiation.cc:240] Negotiation successful
0619 16:47:50.582159 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2408,"mutex_wait_us":253,"spinlock_wait_cycles":10496,"thread_start_us":1348,"threads_started":1}
I20260619 16:47:50.583163 18015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:47:50.571741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:35917 (local address 127.0.0.1:45278)
0619 16:47:50.573558 (+ 1817us) negotiation.cc:107] Waiting for socket to connect
0619 16:47:50.573609 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0619 16:47:50.573819 (+ 210us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:47:50.575431 (+ 1612us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:47:50.575438 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:47:50.575471 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:47:50.576010 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:47:50.576035 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:47:50.577955 (+ 1920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:47:50.577965 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:47:50.580429 (+ 2464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:47:50.580444 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:47:50.582225 (+ 1781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:47:50.582231 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:47:50.582393 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:47:50.582424 (+ 31us) client_negotiation.cc:769] Sending connection context
0619 16:47:50.582487 (+ 63us) client_negotiation.cc:240] Negotiation successful
0619 16:47:50.582612 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1554,"mutex_wait_us":1,"thread_start_us":2330,"threads_started":1}
I20260619 16:47:50.583850 18013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:47:50.571815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45278 (local address 127.0.97.1:35917)
0619 16:47:50.574725 (+ 2910us) server_negotiation.cc:206] Beginning negotiation
0619 16:47:50.574734 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 16:47:50.574763 (+ 29us) server_negotiation.cc:407] Connection header received
0619 16:47:50.574875 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:47:50.574883 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:47:50.575151 (+ 268us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:47:50.576050 (+ 899us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:47:50.576271 (+ 221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:47:50.577817 (+ 1546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:47:50.580595 (+ 2778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:47:50.582015 (+ 1420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:47:50.582731 (+ 716us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:47:50.583199 (+ 468us) server_negotiation.cc:1035] Waiting for connection context
0619 16:47:50.583391 (+ 192us) server_negotiation.cc:299] Negotiation successful
0619 16:47:50.583551 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2741,"thread_start_us":2598,"threads_started":1}
I20260619 16:47:50.585677 18018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:47:50.575509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:42965 (local address 127.0.0.1:48122)
0619 16:47:50.576124 (+ 615us) negotiation.cc:107] Waiting for socket to connect
0619 16:47:50.576150 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0619 16:47:50.576263 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:47:50.577573 (+ 1310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:47:50.577577 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:47:50.577603 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:47:50.578034 (+ 431us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:47:50.578047 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:47:50.580026 (+ 1979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:47:50.580031 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:47:50.581976 (+ 1945us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:47:50.581990 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:47:50.585042 (+ 3052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:47:50.585046 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:47:50.585161 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:47:50.585183 (+ 22us) client_negotiation.cc:769] Sending connection context
0619 16:47:50.585257 (+ 74us) client_negotiation.cc:240] Negotiation successful
0619 16:47:50.585375 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":490,"thread_start_us":344,"threads_started":1}
I20260619 16:47:50.585813 18017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:47:50.575049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48122 (local address 127.0.97.3:42965)
0619 16:47:50.577129 (+ 2080us) server_negotiation.cc:206] Beginning negotiation
0619 16:47:50.577135 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:47:50.577163 (+ 28us) server_negotiation.cc:407] Connection header received
0619 16:47:50.577234 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:47:50.577239 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:47:50.577396 (+ 157us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:47:50.577565 (+ 169us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:47:50.578267 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:47:50.579877 (+ 1610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:47:50.583317 (+ 3440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:47:50.584662 (+ 1345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:47:50.584787 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:47:50.584943 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0619 16:47:50.585316 (+ 373us) server_negotiation.cc:299] Negotiation successful
0619 16:47:50.585469 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1930,"thread_start_us":1762,"threads_started":1}
I20260619 16:47:50.586815 17803 tablet_service.cc:1467] Processing CreateTablet for tablet ade03722fe6140f980b805838cd6a5a9 (DEFAULT_TABLE table=test-table [id=77dd457c8e574e7890db960719d6ca46]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:47:50.588325 17803 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ade03722fe6140f980b805838cd6a5a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:47:50.588678 17669 tablet_service.cc:1467] Processing CreateTablet for tablet ade03722fe6140f980b805838cd6a5a9 (DEFAULT_TABLE table=test-table [id=77dd457c8e574e7890db960719d6ca46]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:47:50.589238 17936 tablet_service.cc:1467] Processing CreateTablet for tablet ade03722fe6140f980b805838cd6a5a9 (DEFAULT_TABLE table=test-table [id=77dd457c8e574e7890db960719d6ca46]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:47:50.590059 17669 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ade03722fe6140f980b805838cd6a5a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:47:50.590550 17936 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ade03722fe6140f980b805838cd6a5a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:47:50.599577 18019 tablet_bootstrap.cc:492] T ade03722fe6140f980b805838cd6a5a9 P ad563589f19344e49bc21900142c3c27: Bootstrap starting.
I20260619 16:47:50.603799 18019 tablet_bootstrap.cc:654] T ade03722fe6140f980b805838cd6a5a9 P ad563589f19344e49bc21900142c3c27: Neither blocks nor log segments found. Creating new log.
I20260619 16:47:50.604202 18021 tablet_bootstrap.cc:492] T ade03722fe6140f980b805838cd6a5a9 P a357c0f5f58d4bd1930248219ca97caf: Bootstrap starting.
I20260619 16:47:50.605283 18019 log.cc:824] T ade03722fe6140f980b805838cd6a5a9 P ad563589f19344e49bc21900142c3c27: Log is configured to *not* fsync() on all Append() calls
I20260619 16:47:50.605656 18020 tablet_bootstrap.cc:492] T ade03722fe6140f980b805838cd6a5a9 P 125454bf52364613932bbe122513f66e: Bootstrap starting.
I20260619 16:47:50.607882 18021 tablet_bootstrap.cc:654] T ade03722fe6140f980b805838cd6a5a9 P a357c0f5f58d4bd1930248219ca97caf: Neither blocks nor log segments found. Creating new log.
I20260619 16:47:50.608572 18020 tablet_bootstrap.cc:654] T ade03722fe6140f980b805838cd6a5a9 P 125454bf52364613932bbe122513f66e: Neither blocks nor log segments found. Creating new log.
I20260619 16:47:50.609177 18021 log.cc:824] T ade03722fe6140f980b805838cd6a5a9 P a357c0f5f58d4bd1930248219ca97caf: Log is configured to *not* fsync() on all Append() calls
I20260619 16:47:50.609645 18020 log.cc:824] T ade03722fe6140f980b805838cd6a5a9 P 125454bf52364613932bbe122513f66e: Log is configured to *not* fsync() on all Append() calls
I20260619 16:47:50.610158 18019 tablet_bootstrap.cc:492] T ade03722fe6140f980b805838cd6a5a9 P ad563589f19344e49bc21900142c3c27: No bootstrap required, opened a new log
I20260619 16:47:50.610468 18019 ts_tablet_manager.cc:1397] T ade03722fe6140f980b805838cd6a5a9 P ad563589f19344e49bc21900142c3c27: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.004s
I20260619 16:47:50.612155 18021 tablet_bootstrap.cc:492] T ade03722fe6140f980b805838cd6a5a9 P a357c0f5f58d4bd1930248219ca97caf: No bootstrap required, opened a new log
I20260619 16:47:50.612416 18021 ts_tablet_manager.cc:1397] T ade03722fe6140f980b805838cd6a5a9 P a357c0f5f58d4bd1930248219ca97caf: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.003s
I20260619 16:47:50.612775 18020 tablet_bootstrap.cc:492] T ade03722fe6140f980b805838cd6a5a9 P 125454bf52364613932bbe122513f66e: No bootstrap required, opened a new log
I20260619 16:47:50.613029 18020 ts_tablet_manager.cc:1397] T ade03722fe6140f980b805838cd6a5a9 P 125454bf52364613932bbe122513f66e: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.003s
I20260619 16:47:50.617903 18019 raft_consensus.cc:348] T ade03722fe6140f980b805838cd6a5a9 P ad563589f19344e49bc21900142c3c27 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a357c0f5f58d4bd1930248219ca97caf" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35917 } } peers { permanent_uuid: "ad563589f19344e49bc21900142c3c27" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44633 } } peers { permanent_uuid: "125454bf52364613932bbe122513f66e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42965 } }
I20260619 16:47:50.618309 18019 raft_consensus.cc:374] T ade03722fe6140f980b805838cd6a5a9 P ad563589f19344e49bc21900142c3c27 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:47:50.618414 18019 raft_consensus.cc:729] T ade03722fe6140f980b805838cd6a5a9 P ad563589f19344e49bc21900142c3c27 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad563589f19344e49bc21900142c3c27, State: Initialized, Role: FOLLOWER
I20260619 16:47:50.619024 18019 consensus_queue.cc:260] T ade03722fe6140f980b805838cd6a5a9 P ad563589f19344e49bc21900142c3c27 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a357c0f5f58d4bd1930248219ca97caf" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35917 } } peers { permanent_uuid: "ad563589f19344e49bc21900142c3c27" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44633 } } peers { permanent_uuid: "125454bf52364613932bbe122513f66e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42965 } }
I20260619 16:47:50.620785 17868 heartbeater.cc:503] Master 127.0.97.62:36589 was elected leader, sending a full tablet report...
I20260619 16:47:50.621506 18019 ts_tablet_manager.cc:1428] T ade03722fe6140f980b805838cd6a5a9 P ad563589f19344e49bc21900142c3c27: Time spent starting tablet: real 0.011s user 0.005s sys 0.005s
I20260619 16:47:50.622663 18021 raft_consensus.cc:348] T ade03722fe6140f980b805838cd6a5a9 P a357c0f5f58d4bd1930248219ca97caf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a357c0f5f58d4bd1930248219ca97caf" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35917 } } peers { permanent_uuid: "ad563589f19344e49bc21900142c3c27" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44633 } } peers { permanent_uuid: "125454bf52364613932bbe122513f66e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42965 } }
I20260619 16:47:50.623095 18021 raft_consensus.cc:374] T ade03722fe6140f980b805838cd6a5a9 P a357c0f5f58d4bd1930248219ca97caf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:47:50.623193 18021 raft_consensus.cc:729] T ade03722fe6140f980b805838cd6a5a9 P a357c0f5f58d4bd1930248219ca97caf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a357c0f5f58d4bd1930248219ca97caf, State: Initialized, Role: FOLLOWER
I20260619 16:47:50.622946 18020 raft_consensus.cc:348] T ade03722fe6140f980b805838cd6a5a9 P 125454bf52364613932bbe122513f66e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a357c0f5f58d4bd1930248219ca97caf" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35917 } } peers { permanent_uuid: "ad563589f19344e49bc21900142c3c27" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44633 } } peers { permanent_uuid: "125454bf52364613932bbe122513f66e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42965 } }
I20260619 16:47:50.623374 18020 raft_consensus.cc:374] T ade03722fe6140f980b805838cd6a5a9 P 125454bf52364613932bbe122513f66e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:47:50.623464 18020 raft_consensus.cc:729] T ade03722fe6140f980b805838cd6a5a9 P 125454bf52364613932bbe122513f66e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 125454bf52364613932bbe122513f66e, State: Initialized, Role: FOLLOWER
I20260619 16:47:50.623875 18021 consensus_queue.cc:260] T ade03722fe6140f980b805838cd6a5a9 P a357c0f5f58d4bd1930248219ca97caf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a357c0f5f58d4bd1930248219ca97caf" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35917 } } peers { permanent_uuid: "ad563589f19344e49bc21900142c3c27" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44633 } } peers { permanent_uuid: "125454bf52364613932bbe122513f66e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42965 } }
I20260619 16:47:50.623960 18020 consensus_queue.cc:260] T ade03722fe6140f980b805838cd6a5a9 P 125454bf52364613932bbe122513f66e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a357c0f5f58d4bd1930248219ca97caf" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35917 } } peers { permanent_uuid: "ad563589f19344e49bc21900142c3c27" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44633 } } peers { permanent_uuid: "125454bf52364613932bbe122513f66e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42965 } }
I20260619 16:47:50.625278 18001 heartbeater.cc:503] Master 127.0.97.62:36589 was elected leader, sending a full tablet report...
I20260619 16:47:50.625608 18021 ts_tablet_manager.cc:1428] T ade03722fe6140f980b805838cd6a5a9 P a357c0f5f58d4bd1930248219ca97caf: Time spent starting tablet: real 0.013s user 0.009s sys 0.005s
I20260619 16:47:50.625722 18020 ts_tablet_manager.cc:1428] T ade03722fe6140f980b805838cd6a5a9 P 125454bf52364613932bbe122513f66e: Time spent starting tablet: real 0.013s user 0.009s sys 0.005s
I20260619 16:47:50.662153 18026 raft_consensus.cc:482] T ade03722fe6140f980b805838cd6a5a9 P a357c0f5f58d4bd1930248219ca97caf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 16:47:50.662441 18026 raft_consensus.cc:504] T ade03722fe6140f980b805838cd6a5a9 P a357c0f5f58d4bd1930248219ca97caf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a357c0f5f58d4bd1930248219ca97caf" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35917 } } peers { permanent_uuid: "ad563589f19344e49bc21900142c3c27" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44633 } } peers { permanent_uuid: "125454bf52364613932bbe122513f66e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42965 } }
I20260619 16:47:50.664229 18026 leader_election.cc:288] T ade03722fe6140f980b805838cd6a5a9 P a357c0f5f58d4bd1930248219ca97caf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ad563589f19344e49bc21900142c3c27 (127.0.97.2:44633), 125454bf52364613932bbe122513f66e (127.0.97.3:42965)
I20260619 16:47:50.671092 18028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:47:50.664539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:42965 (local address 127.0.97.1:32855)
0619 16:47:50.665233 (+ 694us) negotiation.cc:107] Waiting for socket to connect
0619 16:47:50.665253 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 16:47:50.665425 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:47:50.665790 (+ 365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:47:50.665794 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:47:50.665818 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:47:50.666221 (+ 403us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:47:50.666232 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:47:50.667282 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:47:50.667287 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:47:50.668999 (+ 1712us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:47:50.669014 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:47:50.670353 (+ 1339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:47:50.670359 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:47:50.670524 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:47:50.670557 (+ 33us) client_negotiation.cc:769] Sending connection context
0619 16:47:50.670646 (+ 89us) client_negotiation.cc:240] Negotiation successful
0619 16:47:50.670773 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":544,"mutex_wait_us":52,"thread_start_us":346,"threads_started":1}
I20260619 16:47:50.671113 18017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:47:50.664706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:32855 (local address 127.0.97.3:42965)
0619 16:47:50.664889 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0619 16:47:50.664894 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:47:50.665359 (+ 465us) server_negotiation.cc:407] Connection header received
0619 16:47:50.665573 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:47:50.665578 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:47:50.665654 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:47:50.665766 (+ 112us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:47:50.666374 (+ 608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:47:50.667162 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:47:50.669168 (+ 2006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:47:50.670156 (+ 988us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:47:50.670263 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:47:50.670341 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0619 16:47:50.670706 (+ 365us) server_negotiation.cc:299] Negotiation successful
0619 16:47:50.670843 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260619 16:47:50.671499 18016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:47:50.664632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:60381 (local address 127.0.97.2:44633)
0619 16:47:50.664872 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0619 16:47:50.664879 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:47:50.665593 (+ 714us) server_negotiation.cc:407] Connection header received
0619 16:47:50.665776 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:47:50.665781 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:47:50.665854 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:47:50.665959 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:47:50.666570 (+ 611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:47:50.667339 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:47:50.669626 (+ 2287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:47:50.670541 (+ 915us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:47:50.670686 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:47:50.670771 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0619 16:47:50.671153 (+ 382us) server_negotiation.cc:299] Negotiation successful
0619 16:47:50.671277 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260619 16:47:50.672024 18029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:47:50.664633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:44633 (local address 127.0.97.1:60381)
0619 16:47:50.665485 (+ 852us) negotiation.cc:107] Waiting for socket to connect
0619 16:47:50.665511 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0619 16:47:50.665634 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:47:50.666005 (+ 371us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:47:50.666008 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:47:50.666031 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:47:50.666387 (+ 356us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:47:50.666404 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:47:50.667532 (+ 1128us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:47:50.667536 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:47:50.669372 (+ 1836us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:47:50.669388 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:47:50.670784 (+ 1396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:47:50.670796 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:47:50.670980 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:47:50.671014 (+ 34us) client_negotiation.cc:769] Sending connection context
0619 16:47:50.671115 (+ 101us) client_negotiation.cc:240] Negotiation successful
0619 16:47:50.671728 (+ 613us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":753,"mutex_wait_us":36,"thread_start_us":570,"threads_started":1}
I20260619 16:47:50.672034 17956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ade03722fe6140f980b805838cd6a5a9" candidate_uuid: "a357c0f5f58d4bd1930248219ca97caf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125454bf52364613932bbe122513f66e" is_pre_election: true
I20260619 16:47:50.672591 17956 raft_consensus.cc:2454] T ade03722fe6140f980b805838cd6a5a9 P 125454bf52364613932bbe122513f66e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a357c0f5f58d4bd1930248219ca97caf in term 0.
I20260619 16:47:50.672966 17823 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ade03722fe6140f980b805838cd6a5a9" candidate_uuid: "a357c0f5f58d4bd1930248219ca97caf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad563589f19344e49bc21900142c3c27" is_pre_election: true
I20260619 16:47:50.673245 17624 leader_election.cc:302] T ade03722fe6140f980b805838cd6a5a9 P a357c0f5f58d4bd1930248219ca97caf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 125454bf52364613932bbe122513f66e, a357c0f5f58d4bd1930248219ca97caf; no voters:
I20260619 16:47:50.673401 17823 raft_consensus.cc:2454] T ade03722fe6140f980b805838cd6a5a9 P ad563589f19344e49bc21900142c3c27 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a357c0f5f58d4bd1930248219ca97caf in term 0.
I20260619 16:47:50.673607 18026 raft_consensus.cc:2781] T ade03722fe6140f980b805838cd6a5a9 P a357c0f5f58d4bd1930248219ca97caf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 16:47:50.673772 18026 raft_consensus.cc:482] T ade03722fe6140f980b805838cd6a5a9 P a357c0f5f58d4bd1930248219ca97caf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 16:47:50.673893 18026 raft_consensus.cc:3037] T ade03722fe6140f980b805838cd6a5a9 P a357c0f5f58d4bd1930248219ca97caf [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:47:50.676011 18026 raft_consensus.cc:504] T ade03722fe6140f980b805838cd6a5a9 P a357c0f5f58d4bd1930248219ca97caf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a357c0f5f58d4bd1930248219ca97caf" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35917 } } peers { permanent_uuid: "ad563589f19344e49bc21900142c3c27" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44633 } } peers { permanent_uuid: "125454bf52364613932bbe122513f66e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42965 } }
I20260619 16:47:50.676600 18026 leader_election.cc:288] T ade03722fe6140f980b805838cd6a5a9 P a357c0f5f58d4bd1930248219ca97caf [CANDIDATE]: Term 1 election: Requested vote from peers ad563589f19344e49bc21900142c3c27 (127.0.97.2:44633), 125454bf52364613932bbe122513f66e (127.0.97.3:42965)
I20260619 16:47:50.676987 17823 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ade03722fe6140f980b805838cd6a5a9" candidate_uuid: "a357c0f5f58d4bd1930248219ca97caf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad563589f19344e49bc21900142c3c27"
I20260619 16:47:50.677023 17956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ade03722fe6140f980b805838cd6a5a9" candidate_uuid: "a357c0f5f58d4bd1930248219ca97caf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125454bf52364613932bbe122513f66e"
I20260619 16:47:50.677163 17823 raft_consensus.cc:3037] T ade03722fe6140f980b805838cd6a5a9 P ad563589f19344e49bc21900142c3c27 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:47:50.677207 17956 raft_consensus.cc:3037] T ade03722fe6140f980b805838cd6a5a9 P 125454bf52364613932bbe122513f66e [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:47:50.679304 17956 raft_consensus.cc:2454] T ade03722fe6140f980b805838cd6a5a9 P 125454bf52364613932bbe122513f66e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a357c0f5f58d4bd1930248219ca97caf in term 1.
I20260619 16:47:50.679304 17823 raft_consensus.cc:2454] T ade03722fe6140f980b805838cd6a5a9 P ad563589f19344e49bc21900142c3c27 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a357c0f5f58d4bd1930248219ca97caf in term 1.
I20260619 16:47:50.679652 17624 leader_election.cc:302] T ade03722fe6140f980b805838cd6a5a9 P a357c0f5f58d4bd1930248219ca97caf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 125454bf52364613932bbe122513f66e, a357c0f5f58d4bd1930248219ca97caf; no voters:
I20260619 16:47:50.679949 18026 raft_consensus.cc:2781] T ade03722fe6140f980b805838cd6a5a9 P a357c0f5f58d4bd1930248219ca97caf [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:47:50.680363 18026 raft_consensus.cc:686] T ade03722fe6140f980b805838cd6a5a9 P a357c0f5f58d4bd1930248219ca97caf [term 1 LEADER]: Becoming Leader. State: Replica: a357c0f5f58d4bd1930248219ca97caf, State: Running, Role: LEADER
I20260619 16:47:50.680841 18026 consensus_queue.cc:237] T ade03722fe6140f980b805838cd6a5a9 P a357c0f5f58d4bd1930248219ca97caf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a357c0f5f58d4bd1930248219ca97caf" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35917 } } peers { permanent_uuid: "ad563589f19344e49bc21900142c3c27" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44633 } } peers { permanent_uuid: "125454bf52364613932bbe122513f66e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42965 } }
I20260619 16:47:50.684953 17544 catalog_manager.cc:5495] T ade03722fe6140f980b805838cd6a5a9 P a357c0f5f58d4bd1930248219ca97caf reported cstate change: term changed from 0 to 1, leader changed from <none> to a357c0f5f58d4bd1930248219ca97caf (127.0.97.1). New cstate: current_term: 1 leader_uuid: "a357c0f5f58d4bd1930248219ca97caf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a357c0f5f58d4bd1930248219ca97caf" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35917 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ad563589f19344e49bc21900142c3c27" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44633 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "125454bf52364613932bbe122513f66e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42965 } health_report { overall_health: UNKNOWN } } }
I20260619 16:47:50.704748 18013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:47:50.699555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45282 (local address 127.0.97.1:35917)
0619 16:47:50.699724 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0619 16:47:50.699728 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:47:50.699745 (+ 17us) server_negotiation.cc:407] Connection header received
0619 16:47:50.699882 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:47:50.699885 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:47:50.699958 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:47:50.700070 (+ 112us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 16:47:50.700340 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:47:50.701027 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:47:50.702593 (+ 1566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:47:50.702995 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:47:50.703090 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:47:50.703620 (+ 530us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 16:47:50.704090 (+ 470us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 16:47:50.704201 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0619 16:47:50.704358 (+ 157us) server_negotiation.cc:299] Negotiation successful
0619 16:47:50.704437 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260619 16:47:50.712428 17823 raft_consensus.cc:1261] T ade03722fe6140f980b805838cd6a5a9 P ad563589f19344e49bc21900142c3c27 [term 1 FOLLOWER]: Refusing update from remote peer a357c0f5f58d4bd1930248219ca97caf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:47:50.712473 17956 raft_consensus.cc:1261] T ade03722fe6140f980b805838cd6a5a9 P 125454bf52364613932bbe122513f66e [term 1 FOLLOWER]: Refusing update from remote peer a357c0f5f58d4bd1930248219ca97caf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:47:50.713344 18026 consensus_queue.cc:1035] T ade03722fe6140f980b805838cd6a5a9 P a357c0f5f58d4bd1930248219ca97caf [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad563589f19344e49bc21900142c3c27" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44633 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:47:50.713621 18030 consensus_queue.cc:1035] T ade03722fe6140f980b805838cd6a5a9 P a357c0f5f58d4bd1930248219ca97caf [LEADER]: Connected to new peer: Peer: permanent_uuid: "125454bf52364613932bbe122513f66e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:47:50.736756 17545 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39394:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:47:50.737043 17545 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39394:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:47:50.740631 17545 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 258a86eabe5e42368438b37b4c4936fa: Sending DeleteTablet for 3 replicas of tablet ade03722fe6140f980b805838cd6a5a9
I20260619 16:47:50.741829 17936 tablet_service.cc:1514] Processing DeleteTablet for tablet ade03722fe6140f980b805838cd6a5a9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:47:50 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:48122
I20260619 16:47:50.741902 17803 tablet_service.cc:1514] Processing DeleteTablet for tablet ade03722fe6140f980b805838cd6a5a9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:47:50 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:44108
I20260619 16:47:50.741892 17669 tablet_service.cc:1514] Processing DeleteTablet for tablet ade03722fe6140f980b805838cd6a5a9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:47:50 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:45278
I20260619 16:47:50.744765 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu pid 17606
I20260619 16:47:50.754050 18044 tablet_replica.cc:330] T ade03722fe6140f980b805838cd6a5a9 P a357c0f5f58d4bd1930248219ca97caf: stopping tablet replica
I20260619 16:47:50.754619 18044 raft_consensus.cc:2229] T ade03722fe6140f980b805838cd6a5a9 P a357c0f5f58d4bd1930248219ca97caf [term 1 LEADER]: Raft consensus shutting down.
I20260619 16:47:50.755388 18044 raft_consensus.cc:2258] T ade03722fe6140f980b805838cd6a5a9 P a357c0f5f58d4bd1930248219ca97caf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:47:50.756035 18043 tablet_replica.cc:330] T ade03722fe6140f980b805838cd6a5a9 P 125454bf52364613932bbe122513f66e: stopping tablet replica
I20260619 16:47:50.756455 18043 raft_consensus.cc:2229] T ade03722fe6140f980b805838cd6a5a9 P 125454bf52364613932bbe122513f66e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 16:47:50.756975 18043 raft_consensus.cc:2258] T ade03722fe6140f980b805838cd6a5a9 P 125454bf52364613932bbe122513f66e [term 1 FOLLOWER]: Raft consensus is shut down!
Jun 19 16:47:50 dist-test-slave-wvz7 krb5kdc[17500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887668, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260619 16:47:50.759395 18044 ts_tablet_manager.cc:1905] T ade03722fe6140f980b805838cd6a5a9 P a357c0f5f58d4bd1930248219ca97caf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:47:50.760414 18045 tablet_replica.cc:330] T ade03722fe6140f980b805838cd6a5a9 P ad563589f19344e49bc21900142c3c27: stopping tablet replica
I20260619 16:47:50.760710 18045 raft_consensus.cc:2229] T ade03722fe6140f980b805838cd6a5a9 P ad563589f19344e49bc21900142c3c27 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 16:47:50.761107 18045 raft_consensus.cc:2258] T ade03722fe6140f980b805838cd6a5a9 P ad563589f19344e49bc21900142c3c27 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:47:50.761118 18043 ts_tablet_manager.cc:1905] T ade03722fe6140f980b805838cd6a5a9 P 125454bf52364613932bbe122513f66e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:47:50.767807 18043 ts_tablet_manager.cc:1918] T ade03722fe6140f980b805838cd6a5a9 P 125454bf52364613932bbe122513f66e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 16:47:50.767969 18043 log.cc:1196] T ade03722fe6140f980b805838cd6a5a9 P 125454bf52364613932bbe122513f66e: Deleting WAL directory at /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887385903558-388-0/minicluster-data/ts-2/wal/wals/ade03722fe6140f980b805838cd6a5a9
I20260619 16:47:50.768471 18043 ts_tablet_manager.cc:1939] T ade03722fe6140f980b805838cd6a5a9 P 125454bf52364613932bbe122513f66e: Deleting consensus metadata
I20260619 16:47:50.768951 18013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:47:50.746057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45292 (local address 127.0.97.1:35917)
0619 16:47:50.746244 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0619 16:47:50.746249 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:47:50.746272 (+ 23us) server_negotiation.cc:407] Connection header received
0619 16:47:50.746458 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:47:50.746466 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:47:50.746554 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:47:50.746633 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:47:50.748362 (+ 1729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:47:50.749597 (+ 1235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:47:50.751521 (+ 1924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:47:50.752254 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:47:50.752352 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:47:50.759993 (+ 7641us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:47:50.760007 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:47:50.760024 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:47:50.760068 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:47:50.764708 (+ 4640us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:47:50.765623 (+ 915us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:47:50.765631 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:47:50.765635 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:47:50.765867 (+ 232us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:47:50.766331 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:47:50.766336 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:47:50.766339 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:47:50.766791 (+ 452us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:47:50.767097 (+ 306us) server_negotiation.cc:1035] Waiting for connection context
0619 16:47:50.768315 (+ 1218us) server_negotiation.cc:299] Negotiation successful
0619 16:47:50.768575 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260619 16:47:50.769667 17729 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:47:50.770344 17531 catalog_manager.cc:4841] TS 125454bf52364613932bbe122513f66e (127.0.97.3:42965): tablet ade03722fe6140f980b805838cd6a5a9 (table test-table [id=77dd457c8e574e7890db960719d6ca46]) successfully deleted
I20260619 16:47:50.772917 18045 ts_tablet_manager.cc:1905] T ade03722fe6140f980b805838cd6a5a9 P ad563589f19344e49bc21900142c3c27: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:47:50.779695 18045 ts_tablet_manager.cc:1918] T ade03722fe6140f980b805838cd6a5a9 P ad563589f19344e49bc21900142c3c27: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 16:47:50.779872 18045 log.cc:1196] T ade03722fe6140f980b805838cd6a5a9 P ad563589f19344e49bc21900142c3c27: Deleting WAL directory at /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887385903558-388-0/minicluster-data/ts-1/wal/wals/ade03722fe6140f980b805838cd6a5a9
I20260619 16:47:50.780359 18045 ts_tablet_manager.cc:1939] T ade03722fe6140f980b805838cd6a5a9 P ad563589f19344e49bc21900142c3c27: Deleting consensus metadata
I20260619 16:47:50.782205 17530 catalog_manager.cc:4841] TS ad563589f19344e49bc21900142c3c27 (127.0.97.2:44633): tablet ade03722fe6140f980b805838cd6a5a9 (table test-table [id=77dd457c8e574e7890db960719d6ca46]) successfully deleted
I20260619 16:47:50.867573 18044 ts_tablet_manager.cc:1918] T ade03722fe6140f980b805838cd6a5a9 P a357c0f5f58d4bd1930248219ca97caf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 16:47:50.867851 18044 log.cc:1196] T ade03722fe6140f980b805838cd6a5a9 P a357c0f5f58d4bd1930248219ca97caf: Deleting WAL directory at /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887385903558-388-0/minicluster-data/ts-0/wal/wals/ade03722fe6140f980b805838cd6a5a9
I20260619 16:47:50.868297 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu with pid 17606
I20260619 16:47:50.882634 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu pid 17740
W20260619 16:47:50.882756 17531 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.97.1:35917 (error 108)
I20260619 16:47:50.884718 18018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:47:50.884186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:35917 (local address 127.0.0.1:45302)
0619 16:47:50.884405 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0619 16:47:50.884529 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:35917: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101}
W20260619 16:47:50.884953 17531 catalog_manager.cc:4584] TS a357c0f5f58d4bd1930248219ca97caf (127.0.97.1:35917): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ade03722fe6140f980b805838cd6a5a9: Network error: Client connection negotiation failed: client connection to 127.0.97.1:35917: connect: Connection refused (error 111)
Jun 19 16:47:50 dist-test-slave-wvz7 krb5kdc[17500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887668, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260619 16:47:50.897780 18016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:47:50.883484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44124 (local address 127.0.97.2:44633)
0619 16:47:50.883659 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0619 16:47:50.883663 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:47:50.883786 (+ 123us) server_negotiation.cc:407] Connection header received
0619 16:47:50.884035 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:47:50.884041 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:47:50.884148 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:47:50.884258 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:47:50.885769 (+ 1511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:47:50.886508 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:47:50.887759 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:47:50.888147 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:47:50.888258 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:47:50.892395 (+ 4137us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:47:50.892409 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:47:50.892425 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:47:50.892458 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:47:50.895124 (+ 2666us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:47:50.895789 (+ 665us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:47:50.895807 (+ 18us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:47:50.895810 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:47:50.895938 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:47:50.896284 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:47:50.896288 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:47:50.896290 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:47:50.896648 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:47:50.896928 (+ 280us) server_negotiation.cc:1035] Waiting for connection context
0619 16:47:50.897215 (+ 287us) server_negotiation.cc:299] Negotiation successful
0619 16:47:50.897422 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260619 16:47:50.898520 17863 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:47:50.938714 18018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:47:50.938102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:35917 (local address 127.0.0.1:45318)
0619 16:47:50.938345 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0619 16:47:50.938500 (+ 155us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:35917: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20260619 16:47:50.939872 18018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:47:50.939504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:35917 (local address 127.0.0.1:45322)
0619 16:47:50.939651 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0619 16:47:50.939732 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:35917: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260619 16:47:50.989509 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu with pid 17740
I20260619 16:47:51.000607 18018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:47:51.000109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:35917 (local address 127.0.0.1:45336)
0619 16:47:51.000318 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0619 16:47:51.000408 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:35917: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260619 16:47:51.001754 18018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:47:51.001326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:35917 (local address 127.0.0.1:45350)
0619 16:47:51.001490 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0619 16:47:51.001591 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:35917: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260619 16:47:51.002792 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu pid 17873
Jun 19 16:47:51 dist-test-slave-wvz7 krb5kdc[17500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887668, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260619 16:47:51.017901 18017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:47:51.003505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48126 (local address 127.0.97.3:42965)
0619 16:47:51.003680 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0619 16:47:51.003686 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:47:51.003705 (+ 19us) server_negotiation.cc:407] Connection header received
0619 16:47:51.003800 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:47:51.003803 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:47:51.003877 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:47:51.003971 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:47:51.005552 (+ 1581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:47:51.006409 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:47:51.007727 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:47:51.008113 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:47:51.008245 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:47:51.012423 (+ 4178us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:47:51.012440 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:47:51.012456 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:47:51.012495 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:47:51.015145 (+ 2650us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:47:51.015858 (+ 713us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:47:51.015866 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:47:51.015869 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:47:51.016004 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:47:51.016375 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:47:51.016380 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:47:51.016381 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:47:51.016736 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:47:51.017026 (+ 290us) server_negotiation.cc:1035] Waiting for connection context
0619 16:47:51.017302 (+ 276us) server_negotiation.cc:299] Negotiation successful
0619 16:47:51.017520 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260619 16:47:51.018546 17996 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:47:51.082388 18018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:47:51.081927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:35917 (local address 127.0.0.1:45358)
0619 16:47:51.082118 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0619 16:47:51.082214 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:35917: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260619 16:47:51.083405 18018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:47:51.082994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:35917 (local address 127.0.0.1:45360)
0619 16:47:51.083219 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0619 16:47:51.083289 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:35917: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95}
I20260619 16:47:51.106405 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu with pid 17873
I20260619 16:47:51.119251 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu pid 17513
I20260619 16:47:51.120309 17575 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:47:51.229539 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu with pid 17513
2026-06-19T16:47:51Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (3979 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-06-19T16:47:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T16:47:51Z Disabled control of system clock
I20260619 16:47:51.298058 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu
/tmp/dist-test-taskEs2EH8/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781887385903558-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781887385903558-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781887385903558-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781887385903558-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:46581
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:35135
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:46581
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 16:47:51.613142 18055 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:47:51.613408 18055 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:47:51.613493 18055 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:47:51.620662 18055 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:47:51.620769 18055 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:47:51.620805 18055 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:47:51.620862 18055 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:47:51.620903 18055 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:47:51.630137 18055 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35135
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781887385903558-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781887385903558-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:46581
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:46581
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781887385903558-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781887385903558-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:47:51.631351 18055 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:47:51.633023 18055 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:47:51.643800 18060 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:47:51.643869 18063 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:47:51.643891 18061 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:47:51.645028 18055 server_base.cc:1034] running on GCE node
I20260619 16:47:51.645845 18055 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:47:51.647265 18055 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:47:51.648466 18055 hybrid_clock.cc:648] HybridClock initialized: now 1781887671648414 us; error 48 us; skew 500 ppm
I20260619 16:47:51.648869 18055 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 16:47:51.651053 18055 webserver.cc:460] Webserver started at http://127.0.97.62:41917/ using document root <none> and password file <none>
I20260619 16:47:51.651698 18055 fs_manager.cc:359] Metadata directory not provided
I20260619 16:47:51.651794 18055 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:47:51.652088 18055 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:47:51.654907 18055 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781887385903558-388-0/minicluster-data/master-0/data/instance:
uuid: "003e2cf86ec44b63be7819e7085826e6"
format_stamp: "Formatted at 2026-06-19 16:47:51 on dist-test-slave-wvz7"
server_key: "a5a5962653f8993f8d14ebaee42863c3"
server_key_iv: "a49a5dd661dfbafd117b3b9e98a7ff68"
server_key_version: "encryptionkey@0"
I20260619 16:47:51.655669 18055 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781887385903558-388-0/minicluster-data/master-0/wal/instance:
uuid: "003e2cf86ec44b63be7819e7085826e6"
format_stamp: "Formatted at 2026-06-19 16:47:51 on dist-test-slave-wvz7"
server_key: "a5a5962653f8993f8d14ebaee42863c3"
server_key_iv: "a49a5dd661dfbafd117b3b9e98a7ff68"
server_key_version: "encryptionkey@0"
I20260619 16:47:51.660548 18055 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260619 16:47:51.663810 18069 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:47:51.665061 18055 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260619 16:47:51.665213 18055 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781887385903558-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781887385903558-388-0/minicluster-data/master-0/wal
uuid: "003e2cf86ec44b63be7819e7085826e6"
format_stamp: "Formatted at 2026-06-19 16:47:51 on dist-test-slave-wvz7"
server_key: "a5a5962653f8993f8d14ebaee42863c3"
server_key_iv: "a49a5dd661dfbafd117b3b9e98a7ff68"
server_key_version: "encryptionkey@0"
I20260619 16:47:51.665405 18055 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781887385903558-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781887385903558-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781887385903558-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:47:51.684301 18055 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:47:51.685144 18055 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:47:51.685420 18055 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:47:51.702683 18055 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:46581
I20260619 16:47:51.702701 18120 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:46581 every 8 connection(s)
I20260619 16:47:51.703985 18055 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781887385903558-388-0/minicluster-data/master-0/data/info.pb
I20260619 16:47:51.707597 18121 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:47:51.709110 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu as pid 18055
I20260619 16:47:51.709259 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781887385903558-388-0/minicluster-data/master-0/wal/instance
I20260619 16:47:51.709676 388 external_mini_cluster.cc:1428] Setting key 8f8fbc0c79d2b315a73ec184ce0249e9
I20260619 16:47:51.719715 18121 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 003e2cf86ec44b63be7819e7085826e6: Bootstrap starting.
I20260619 16:47:51.722433 18124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:47:51.711503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34244 (local address 127.0.97.62:46581)
0619 16:47:51.712475 (+ 972us) server_negotiation.cc:206] Beginning negotiation
0619 16:47:51.712486 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 16:47:51.712525 (+ 39us) server_negotiation.cc:407] Connection header received
0619 16:47:51.714146 (+ 1621us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:47:51.714168 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:47:51.714644 (+ 476us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:47:51.714971 (+ 327us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:47:51.715517 (+ 546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:47:51.716743 (+ 1226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:47:51.718122 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:47:51.718883 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:47:51.719056 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:47:51.720212 (+ 1156us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:47:51.720219 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:47:51.720233 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:47:51.720754 (+ 521us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:47:51.720831 (+ 77us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:47:51.720852 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:47:51.720952 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0619 16:47:51.721180 (+ 228us) server_negotiation.cc:299] Negotiation successful
0619 16:47:51.721463 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":551,"thread_start_us":338,"threads_started":1}
I20260619 16:47:51.722981 18121 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 003e2cf86ec44b63be7819e7085826e6: Neither blocks nor log segments found. Creating new log.
I20260619 16:47:51.724136 18121 log.cc:824] T 00000000000000000000000000000000 P 003e2cf86ec44b63be7819e7085826e6: Log is configured to *not* fsync() on all Append() calls
I20260619 16:47:51.726706 18121 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 003e2cf86ec44b63be7819e7085826e6: No bootstrap required, opened a new log
I20260619 16:47:51.733467 18121 raft_consensus.cc:348] T 00000000000000000000000000000000 P 003e2cf86ec44b63be7819e7085826e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "003e2cf86ec44b63be7819e7085826e6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46581 } }
I20260619 16:47:51.733841 18121 raft_consensus.cc:374] T 00000000000000000000000000000000 P 003e2cf86ec44b63be7819e7085826e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:47:51.733943 18121 raft_consensus.cc:729] T 00000000000000000000000000000000 P 003e2cf86ec44b63be7819e7085826e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 003e2cf86ec44b63be7819e7085826e6, State: Initialized, Role: FOLLOWER
I20260619 16:47:51.734503 18121 consensus_queue.cc:260] T 00000000000000000000000000000000 P 003e2cf86ec44b63be7819e7085826e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "003e2cf86ec44b63be7819e7085826e6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46581 } }
I20260619 16:47:51.734726 18121 raft_consensus.cc:388] T 00000000000000000000000000000000 P 003e2cf86ec44b63be7819e7085826e6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:47:51.734848 18121 raft_consensus.cc:482] T 00000000000000000000000000000000 P 003e2cf86ec44b63be7819e7085826e6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:47:51.734995 18121 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 003e2cf86ec44b63be7819e7085826e6 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:47:51.737171 18121 raft_consensus.cc:504] T 00000000000000000000000000000000 P 003e2cf86ec44b63be7819e7085826e6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "003e2cf86ec44b63be7819e7085826e6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46581 } }
I20260619 16:47:51.737623 18121 leader_election.cc:302] T 00000000000000000000000000000000 P 003e2cf86ec44b63be7819e7085826e6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 003e2cf86ec44b63be7819e7085826e6; no voters:
I20260619 16:47:51.738168 18121 leader_election.cc:288] T 00000000000000000000000000000000 P 003e2cf86ec44b63be7819e7085826e6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 16:47:51.738299 18126 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 003e2cf86ec44b63be7819e7085826e6 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:47:51.738862 18126 raft_consensus.cc:686] T 00000000000000000000000000000000 P 003e2cf86ec44b63be7819e7085826e6 [term 1 LEADER]: Becoming Leader. State: Replica: 003e2cf86ec44b63be7819e7085826e6, State: Running, Role: LEADER
I20260619 16:47:51.739384 18126 consensus_queue.cc:237] T 00000000000000000000000000000000 P 003e2cf86ec44b63be7819e7085826e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "003e2cf86ec44b63be7819e7085826e6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46581 } }
I20260619 16:47:51.740218 18121 sys_catalog.cc:564] T 00000000000000000000000000000000 P 003e2cf86ec44b63be7819e7085826e6 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:47:51.741866 18128 sys_catalog.cc:455] T 00000000000000000000000000000000 P 003e2cf86ec44b63be7819e7085826e6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "003e2cf86ec44b63be7819e7085826e6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "003e2cf86ec44b63be7819e7085826e6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46581 } } }
I20260619 16:47:51.742277 18128 sys_catalog.cc:458] T 00000000000000000000000000000000 P 003e2cf86ec44b63be7819e7085826e6 [sys.catalog]: This master's current role is: LEADER
I20260619 16:47:51.742689 18127 sys_catalog.cc:455] T 00000000000000000000000000000000 P 003e2cf86ec44b63be7819e7085826e6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 003e2cf86ec44b63be7819e7085826e6. Latest consensus state: current_term: 1 leader_uuid: "003e2cf86ec44b63be7819e7085826e6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "003e2cf86ec44b63be7819e7085826e6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46581 } } }
I20260619 16:47:51.743022 18127 sys_catalog.cc:458] T 00000000000000000000000000000000 P 003e2cf86ec44b63be7819e7085826e6 [sys.catalog]: This master's current role is: LEADER
I20260619 16:47:51.744407 18134 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:47:51.748617 18134 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:47:51.756839 18134 catalog_manager.cc:1347] Generated new cluster ID: aab5c45eedde4c469b5f06b28a0de9b1
I20260619 16:47:51.756981 18134 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:47:51.787225 18134 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 16:47:51.789043 18134 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:47:51.803687 18134 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 003e2cf86ec44b63be7819e7085826e6: Generated new TSK 0
I20260619 16:47:51.804733 18134 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 16:47:51.813057 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu
/tmp/dist-test-taskEs2EH8/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781887385903558-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781887385903558-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781887385903558-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781887385903558-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46581
--builtin_ntp_servers=127.0.97.20:35135
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 16:47:52.136299 18145 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:47:52.136588 18145 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:47:52.136673 18145 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:47:52.143946 18145 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:47:52.144062 18145 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:47:52.144199 18145 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260619 16:47:52.153221 18145 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35135
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781887385903558-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781887385903558-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781887385903558-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46581
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781887385903558-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:47:52.154536 18145 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:47:52.156287 18145 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:47:52.168756 18150 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:47:52.168888 18151 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:47:52.168884 18153 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:47:52.169704 18145 server_base.cc:1034] running on GCE node
I20260619 16:47:52.170392 18145 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:47:52.171319 18145 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:47:52.172523 18145 hybrid_clock.cc:648] HybridClock initialized: now 1781887672172455 us; error 57 us; skew 500 ppm
I20260619 16:47:52.172847 18145 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 16:47:52.174995 18145 webserver.cc:460] Webserver started at http://127.0.97.1:39229/ using document root <none> and password file <none>
I20260619 16:47:52.175673 18145 fs_manager.cc:359] Metadata directory not provided
I20260619 16:47:52.175769 18145 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:47:52.176059 18145 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:47:52.178850 18145 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781887385903558-388-0/minicluster-data/ts-0/data/instance:
uuid: "61cdcb94fbae4942b904f01024e847ee"
format_stamp: "Formatted at 2026-06-19 16:47:52 on dist-test-slave-wvz7"
server_key: "048076e32ba14d6a42e7d20f016178d6"
server_key_iv: "4473baab9bd9d9a6c950d15e1e9290ec"
server_key_version: "encryptionkey@0"
I20260619 16:47:52.179581 18145 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781887385903558-388-0/minicluster-data/ts-0/wal/instance:
uuid: "61cdcb94fbae4942b904f01024e847ee"
format_stamp: "Formatted at 2026-06-19 16:47:52 on dist-test-slave-wvz7"
server_key: "048076e32ba14d6a42e7d20f016178d6"
server_key_iv: "4473baab9bd9d9a6c950d15e1e9290ec"
server_key_version: "encryptionkey@0"
I20260619 16:47:52.184540 18145 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 16:47:52.187929 18159 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:47:52.189366 18145 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260619 16:47:52.189518 18145 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781887385903558-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781887385903558-388-0/minicluster-data/ts-0/wal
uuid: "61cdcb94fbae4942b904f01024e847ee"
format_stamp: "Formatted at 2026-06-19 16:47:52 on dist-test-slave-wvz7"
server_key: "048076e32ba14d6a42e7d20f016178d6"
server_key_iv: "4473baab9bd9d9a6c950d15e1e9290ec"
server_key_version: "encryptionkey@0"
I20260619 16:47:52.189723 18145 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781887385903558-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781887385903558-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781887385903558-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:47:52.205324 18145 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:47:52.206246 18145 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:47:52.206508 18145 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:47:52.207414 18145 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:47:52.208887 18145 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:47:52.208981 18145 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:47:52.209090 18145 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:47:52.209156 18145 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:47:52.239619 18145 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:45595
I20260619 16:47:52.239660 18271 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:45595 every 8 connection(s)
I20260619 16:47:52.240885 18145 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781887385903558-388-0/minicluster-data/ts-0/data/info.pb
I20260619 16:47:52.245221 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu as pid 18145
I20260619 16:47:52.245360 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781887385903558-388-0/minicluster-data/ts-0/wal/instance
I20260619 16:47:52.245819 388 external_mini_cluster.cc:1428] Setting key 2eaa5cc9018b674068cdf8252b4b52fc
I20260619 16:47:52.248596 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu
/tmp/dist-test-taskEs2EH8/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781887385903558-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781887385903558-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781887385903558-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781887385903558-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46581
--builtin_ntp_servers=127.0.97.20:35135
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260619 16:47:52.253567 18274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:47:52.243245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:37293 (local address 127.0.97.62:46581)
0619 16:47:52.243764 (+ 519us) server_negotiation.cc:206] Beginning negotiation
0619 16:47:52.243771 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:47:52.244895 (+ 1124us) server_negotiation.cc:407] Connection header received
0619 16:47:52.246948 (+ 2053us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:47:52.246953 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:47:52.247016 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:47:52.247126 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:47:52.248367 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:47:52.249140 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:47:52.250880 (+ 1740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:47:52.251281 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:47:52.251391 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:47:52.252754 (+ 1363us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:47:52.252759 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:47:52.252763 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:47:52.252782 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:47:52.252822 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:47:52.252835 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:47:52.252907 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0619 16:47:52.253114 (+ 207us) server_negotiation.cc:299] Negotiation successful
0619 16:47:52.253213 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":388,"thread_start_us":592,"threads_started":1}
I20260619 16:47:52.254585 18275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:47:52.243834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46581 (local address 127.0.97.1:37293)
0619 16:47:52.244759 (+ 925us) negotiation.cc:107] Waiting for socket to connect
0619 16:47:52.244796 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0619 16:47:52.246673 (+ 1877us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:47:52.247375 (+ 702us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:47:52.247386 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:47:52.247515 (+ 129us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:47:52.248203 (+ 688us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:47:52.248221 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:47:52.249288 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:47:52.249291 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:47:52.250741 (+ 1450us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:47:52.250754 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:47:52.251490 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:47:52.251497 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:47:52.251784 (+ 287us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:47:52.252443 (+ 659us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 16:47:52.252445 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:47:52.252485 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 16:47:52.252569 (+ 84us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:47:52.252920 (+ 351us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:47:52.252932 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:47:52.252940 (+ 8us) client_negotiation.cc:769] Sending connection context
0619 16:47:52.253104 (+ 164us) client_negotiation.cc:240] Negotiation successful
0619 16:47:52.253510 (+ 406us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":441,"thread_start_us":268,"threads_started":1}
I20260619 16:47:52.256052 18272 heartbeater.cc:347] Connected to a master server at 127.0.97.62:46581
I20260619 16:47:52.256352 18272 heartbeater.cc:464] Registering TS with master...
I20260619 16:47:52.257203 18272 heartbeater.cc:511] Master 127.0.97.62:46581 requested a full tablet report, sending...
I20260619 16:47:52.259275 18086 ts_manager.cc:194] Registered new tserver with Master: 61cdcb94fbae4942b904f01024e847ee (127.0.97.1:45595)
I20260619 16:47:52.260706 18086 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.1:37293
W20260619 16:47:52.563373 18276 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:47:52.563665 18276 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:47:52.563771 18276 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:47:52.571141 18276 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:47:52.571239 18276 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:47:52.571373 18276 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260619 16:47:52.580500 18276 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35135
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781887385903558-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781887385903558-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781887385903558-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46581
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781887385903558-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:47:52.581840 18276 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:47:52.583552 18276 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:47:52.596354 18281 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:47:52.596287 18282 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:47:52.596328 18284 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:47:52.597097 18276 server_base.cc:1034] running on GCE node
I20260619 16:47:52.597689 18276 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:47:52.599211 18276 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:47:52.600425 18276 hybrid_clock.cc:648] HybridClock initialized: now 1781887672600370 us; error 47 us; skew 500 ppm
I20260619 16:47:52.600771 18276 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 16:47:52.603052 18276 webserver.cc:460] Webserver started at http://127.0.97.2:42637/ using document root <none> and password file <none>
I20260619 16:47:52.603706 18276 fs_manager.cc:359] Metadata directory not provided
I20260619 16:47:52.603789 18276 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:47:52.604071 18276 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:47:52.606868 18276 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781887385903558-388-0/minicluster-data/ts-1/data/instance:
uuid: "d186a9b6f13c4bcca28612936da1f89d"
format_stamp: "Formatted at 2026-06-19 16:47:52 on dist-test-slave-wvz7"
server_key: "da3cc80f423d38ed935298cb980a5911"
server_key_iv: "9f93739d5f4336369b71d3479307a8f6"
server_key_version: "encryptionkey@0"
I20260619 16:47:52.607595 18276 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781887385903558-388-0/minicluster-data/ts-1/wal/instance:
uuid: "d186a9b6f13c4bcca28612936da1f89d"
format_stamp: "Formatted at 2026-06-19 16:47:52 on dist-test-slave-wvz7"
server_key: "da3cc80f423d38ed935298cb980a5911"
server_key_iv: "9f93739d5f4336369b71d3479307a8f6"
server_key_version: "encryptionkey@0"
I20260619 16:47:52.612636 18276 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260619 16:47:52.616207 18290 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:47:52.617520 18276 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260619 16:47:52.617683 18276 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781887385903558-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781887385903558-388-0/minicluster-data/ts-1/wal
uuid: "d186a9b6f13c4bcca28612936da1f89d"
format_stamp: "Formatted at 2026-06-19 16:47:52 on dist-test-slave-wvz7"
server_key: "da3cc80f423d38ed935298cb980a5911"
server_key_iv: "9f93739d5f4336369b71d3479307a8f6"
server_key_version: "encryptionkey@0"
I20260619 16:47:52.617950 18276 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781887385903558-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781887385903558-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781887385903558-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:47:52.633050 18276 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:47:52.634006 18276 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:47:52.634298 18276 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:47:52.635327 18276 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:47:52.636926 18276 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:47:52.637035 18276 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:47:52.637154 18276 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:47:52.637233 18276 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:47:52.667428 18276 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:42429
I20260619 16:47:52.667483 18402 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:42429 every 8 connection(s)
I20260619 16:47:52.668746 18276 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781887385903558-388-0/minicluster-data/ts-1/data/info.pb
I20260619 16:47:52.672643 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu as pid 18276
I20260619 16:47:52.672807 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781887385903558-388-0/minicluster-data/ts-1/wal/instance
I20260619 16:47:52.673321 388 external_mini_cluster.cc:1428] Setting key f016e225681712c7b978b2e1b220733b
I20260619 16:47:52.676208 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu
/tmp/dist-test-taskEs2EH8/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781887385903558-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781887385903558-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781887385903558-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781887385903558-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46581
--builtin_ntp_servers=127.0.97.20:35135
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260619 16:47:52.681670 18274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:47:52.671252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:51391 (local address 127.0.97.62:46581)
0619 16:47:52.671405 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0619 16:47:52.671410 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:47:52.672946 (+ 1536us) server_negotiation.cc:407] Connection header received
0619 16:47:52.675143 (+ 2197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:47:52.675148 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:47:52.675240 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:47:52.675311 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:47:52.676475 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:47:52.677471 (+ 996us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:47:52.679100 (+ 1629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:47:52.679622 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:47:52.679735 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:47:52.680969 (+ 1234us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:47:52.680975 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:47:52.680979 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:47:52.681000 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:47:52.681038 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:47:52.681051 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:47:52.681127 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0619 16:47:52.681296 (+ 169us) server_negotiation.cc:299] Negotiation successful
0619 16:47:52.681388 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260619 16:47:52.682786 18405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:47:52.671756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46581 (local address 127.0.97.2:51391)
0619 16:47:52.672792 (+ 1036us) negotiation.cc:107] Waiting for socket to connect
0619 16:47:52.672840 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0619 16:47:52.674840 (+ 2000us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:47:52.675539 (+ 699us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:47:52.675547 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:47:52.675640 (+ 93us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:47:52.676192 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:47:52.676208 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:47:52.677649 (+ 1441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:47:52.677653 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:47:52.678953 (+ 1300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:47:52.678966 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:47:52.679811 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:47:52.679815 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:47:52.680033 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:47:52.680671 (+ 638us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 16:47:52.680673 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:47:52.680712 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 16:47:52.680789 (+ 77us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:47:52.681129 (+ 340us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:47:52.681135 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:47:52.681141 (+ 6us) client_negotiation.cc:769] Sending connection context
0619 16:47:52.681270 (+ 129us) client_negotiation.cc:240] Negotiation successful
0619 16:47:52.681582 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":447,"thread_start_us":285,"threads_started":1}
I20260619 16:47:52.684264 18403 heartbeater.cc:347] Connected to a master server at 127.0.97.62:46581
I20260619 16:47:52.684544 18403 heartbeater.cc:464] Registering TS with master...
I20260619 16:47:52.685372 18403 heartbeater.cc:511] Master 127.0.97.62:46581 requested a full tablet report, sending...
I20260619 16:47:52.686846 18086 ts_manager.cc:194] Registered new tserver with Master: d186a9b6f13c4bcca28612936da1f89d (127.0.97.2:42429)
I20260619 16:47:52.687739 18086 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.2:51391
W20260619 16:47:52.987056 18406 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:47:52.987334 18406 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:47:52.987440 18406 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:47:52.994802 18406 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:47:52.994905 18406 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:47:52.995054 18406 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260619 16:47:53.004076 18406 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35135
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781887385903558-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781887385903558-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781887385903558-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46581
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781887385903558-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:47:53.005302 18406 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:47:53.007002 18406 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:47:53.019294 18411 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:47:53.019333 18414 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:47:53.019366 18412 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:47:53.020112 18406 server_base.cc:1034] running on GCE node
I20260619 16:47:53.020634 18406 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:47:53.021564 18406 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:47:53.022783 18406 hybrid_clock.cc:648] HybridClock initialized: now 1781887673022737 us; error 37 us; skew 500 ppm
I20260619 16:47:53.023108 18406 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 16:47:53.025293 18406 webserver.cc:460] Webserver started at http://127.0.97.3:43025/ using document root <none> and password file <none>
I20260619 16:47:53.026010 18406 fs_manager.cc:359] Metadata directory not provided
I20260619 16:47:53.026113 18406 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:47:53.026405 18406 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:47:53.029145 18406 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781887385903558-388-0/minicluster-data/ts-2/data/instance:
uuid: "73d867d2d584466fa4d8c22fabcdfccb"
format_stamp: "Formatted at 2026-06-19 16:47:53 on dist-test-slave-wvz7"
server_key: "1c21ef71887cd175b282622f24e427e4"
server_key_iv: "7189239850b944026cecb9718e96c990"
server_key_version: "encryptionkey@0"
I20260619 16:47:53.029955 18406 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781887385903558-388-0/minicluster-data/ts-2/wal/instance:
uuid: "73d867d2d584466fa4d8c22fabcdfccb"
format_stamp: "Formatted at 2026-06-19 16:47:53 on dist-test-slave-wvz7"
server_key: "1c21ef71887cd175b282622f24e427e4"
server_key_iv: "7189239850b944026cecb9718e96c990"
server_key_version: "encryptionkey@0"
I20260619 16:47:53.034854 18406 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260619 16:47:53.038311 18420 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:47:53.039754 18406 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260619 16:47:53.039904 18406 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781887385903558-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781887385903558-388-0/minicluster-data/ts-2/wal
uuid: "73d867d2d584466fa4d8c22fabcdfccb"
format_stamp: "Formatted at 2026-06-19 16:47:53 on dist-test-slave-wvz7"
server_key: "1c21ef71887cd175b282622f24e427e4"
server_key_iv: "7189239850b944026cecb9718e96c990"
server_key_version: "encryptionkey@0"
I20260619 16:47:53.040110 18406 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781887385903558-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781887385903558-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781887385903558-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:47:53.052721 18406 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:47:53.053467 18406 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:47:53.053720 18406 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:47:53.054639 18406 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:47:53.056052 18406 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:47:53.056145 18406 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:47:53.056247 18406 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:47:53.056313 18406 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:47:53.085318 18406 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:42709
I20260619 16:47:53.085358 18532 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:42709 every 8 connection(s)
I20260619 16:47:53.086598 18406 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781887385903558-388-0/minicluster-data/ts-2/data/info.pb
I20260619 16:47:53.090852 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu as pid 18406
I20260619 16:47:53.091073 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781887385903558-388-0/minicluster-data/ts-2/wal/instance
I20260619 16:47:53.091630 388 external_mini_cluster.cc:1428] Setting key 360bc55ba256fb5f98a848050ece0dce
I20260619 16:47:53.099690 18274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:47:53.088984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:56691 (local address 127.0.97.62:46581)
0619 16:47:53.089167 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0619 16:47:53.089171 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:47:53.090794 (+ 1623us) server_negotiation.cc:407] Connection header received
0619 16:47:53.093161 (+ 2367us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:47:53.093166 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:47:53.093249 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:47:53.093353 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:47:53.094793 (+ 1440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:47:53.095535 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:47:53.097107 (+ 1572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:47:53.097550 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:47:53.097655 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:47:53.099009 (+ 1354us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:47:53.099014 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:47:53.099017 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:47:53.099034 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:47:53.099067 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:47:53.099079 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:47:53.099154 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0619 16:47:53.099363 (+ 209us) server_negotiation.cc:299] Negotiation successful
0619 16:47:53.099443 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260619 16:47:53.100654 18535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:47:53.089493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46581 (local address 127.0.97.3:56691)
0619 16:47:53.090605 (+ 1112us) negotiation.cc:107] Waiting for socket to connect
0619 16:47:53.090657 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0619 16:47:53.092852 (+ 2195us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:47:53.093837 (+ 985us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:47:53.093846 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:47:53.093943 (+ 97us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:47:53.094654 (+ 711us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:47:53.094675 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:47:53.095699 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:47:53.095704 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:47:53.096958 (+ 1254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:47:53.096972 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:47:53.097785 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:47:53.097793 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:47:53.098009 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:47:53.098687 (+ 678us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 16:47:53.098690 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:47:53.098735 (+ 45us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 16:47:53.098812 (+ 77us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:47:53.099168 (+ 356us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:47:53.099176 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:47:53.099183 (+ 7us) client_negotiation.cc:769] Sending connection context
0619 16:47:53.099315 (+ 132us) client_negotiation.cc:240] Negotiation successful
0619 16:47:53.099643 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":527,"thread_start_us":344,"threads_started":1}
I20260619 16:47:53.101908 18533 heartbeater.cc:347] Connected to a master server at 127.0.97.62:46581
I20260619 16:47:53.102207 18533 heartbeater.cc:464] Registering TS with master...
I20260619 16:47:53.102892 18533 heartbeater.cc:511] Master 127.0.97.62:46581 requested a full tablet report, sending...
I20260619 16:47:53.104148 18086 ts_manager.cc:194] Registered new tserver with Master: 73d867d2d584466fa4d8c22fabcdfccb (127.0.97.3:42709)
I20260619 16:47:53.105051 18086 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.3:56691
I20260619 16:47:53.107023 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 16:47:53.116781 18274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:47:53.111117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60778 (local address 127.0.97.62:46581)
0619 16:47:53.111283 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0619 16:47:53.111290 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:47:53.111699 (+ 409us) server_negotiation.cc:407] Connection header received
0619 16:47:53.111869 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:47:53.111873 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:47:53.111947 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:47:53.112027 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:47:53.112493 (+ 466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:47:53.113157 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:47:53.114563 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:47:53.114939 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:47:53.115057 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:47:53.116124 (+ 1067us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:47:53.116130 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:47:53.116133 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:47:53.116152 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:47:53.116183 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:47:53.116196 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:47:53.116269 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0619 16:47:53.116395 (+ 126us) server_negotiation.cc:299] Negotiation successful
0619 16:47:53.116486 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260619 16:47:53.120442 18086 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60778:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 16:47:53.127305 18086 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 16:47:53.161723 18547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:47:53.138961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45595 (local address 127.0.0.1:34756)
0619 16:47:53.152503 (+ 13542us) negotiation.cc:107] Waiting for socket to connect
0619 16:47:53.152556 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0619 16:47:53.152752 (+ 196us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:47:53.154011 (+ 1259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:47:53.154018 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:47:53.154048 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:47:53.154453 (+ 405us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:47:53.154479 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:47:53.156320 (+ 1841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:47:53.156325 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:47:53.158769 (+ 2444us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:47:53.158783 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:47:53.160852 (+ 2069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:47:53.160859 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:47:53.161059 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:47:53.161119 (+ 60us) client_negotiation.cc:769] Sending connection context
0619 16:47:53.161236 (+ 117us) client_negotiation.cc:240] Negotiation successful
0619 16:47:53.161406 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13226,"thread_start_us":15184,"threads_started":1}
I20260619 16:47:53.161985 18546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:47:53.139010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34756 (local address 127.0.97.1:45595)
0619 16:47:53.151870 (+ 12860us) server_negotiation.cc:206] Beginning negotiation
0619 16:47:53.151882 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 16:47:53.153145 (+ 1263us) server_negotiation.cc:407] Connection header received
0619 16:47:53.153263 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:47:53.153273 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:47:53.153554 (+ 281us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:47:53.153760 (+ 206us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:47:53.154666 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:47:53.156168 (+ 1502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:47:53.158969 (+ 2801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:47:53.160379 (+ 1410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:47:53.160488 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:47:53.160726 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0619 16:47:53.161323 (+ 597us) server_negotiation.cc:299] Negotiation successful
0619 16:47:53.161496 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12662,"thread_start_us":13888,"threads_started":1}
I20260619 16:47:53.163276 18549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:47:53.140872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:42429 (local address 127.0.0.1:49986)
0619 16:47:53.156274 (+ 15402us) negotiation.cc:107] Waiting for socket to connect
0619 16:47:53.156289 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 16:47:53.156423 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:47:53.156988 (+ 565us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:47:53.156993 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:47:53.157017 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:47:53.157409 (+ 392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:47:53.157422 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:47:53.159181 (+ 1759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:47:53.159186 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:47:53.160822 (+ 1636us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:47:53.160840 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:47:53.162481 (+ 1641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:47:53.162487 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:47:53.162712 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:47:53.162748 (+ 36us) client_negotiation.cc:769] Sending connection context
0619 16:47:53.162823 (+ 75us) client_negotiation.cc:240] Negotiation successful
0619 16:47:53.162940 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15294,"mutex_wait_us":42,"thread_start_us":15147,"threads_started":1}
I20260619 16:47:53.163403 18550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:47:53.141537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49986 (local address 127.0.97.2:42429)
0619 16:47:53.155850 (+ 14313us) server_negotiation.cc:206] Beginning negotiation
0619 16:47:53.155860 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 16:47:53.156416 (+ 556us) server_negotiation.cc:407] Connection header received
0619 16:47:53.156582 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:47:53.156590 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:47:53.156807 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:47:53.157011 (+ 204us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:47:53.157593 (+ 582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:47:53.159060 (+ 1467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:47:53.161026 (+ 1966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:47:53.162308 (+ 1282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:47:53.162423 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:47:53.162690 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0619 16:47:53.162884 (+ 194us) server_negotiation.cc:299] Negotiation successful
0619 16:47:53.163050 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14159,"thread_start_us":14032,"threads_started":1}
I20260619 16:47:53.163753 18545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:47:53.139064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:42709 (local address 127.0.0.1:34392)
0619 16:47:53.153945 (+ 14881us) negotiation.cc:107] Waiting for socket to connect
0619 16:47:53.153978 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0619 16:47:53.154080 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:47:53.156246 (+ 2166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:47:53.156251 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:47:53.156277 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:47:53.156780 (+ 503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:47:53.156790 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:47:53.158363 (+ 1573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:47:53.158367 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:47:53.160260 (+ 1893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:47:53.160273 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:47:53.162635 (+ 2362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:47:53.162642 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:47:53.162833 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:47:53.162864 (+ 31us) client_negotiation.cc:769] Sending connection context
0619 16:47:53.162942 (+ 78us) client_negotiation.cc:240] Negotiation successful
0619 16:47:53.163058 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14766,"mutex_wait_us":5,"spinlock_wait_cycles":896,"thread_start_us":12980,"threads_started":1}
I20260619 16:47:53.164559 18548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:47:53.139595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34392 (local address 127.0.97.3:42709)
0619 16:47:53.155557 (+ 15962us) server_negotiation.cc:206] Beginning negotiation
0619 16:47:53.155566 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 16:47:53.155598 (+ 32us) server_negotiation.cc:407] Connection header received
0619 16:47:53.155687 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:47:53.155694 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:47:53.155938 (+ 244us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:47:53.156117 (+ 179us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:47:53.156941 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:47:53.158246 (+ 1305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:47:53.160494 (+ 2248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:47:53.162047 (+ 1553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:47:53.162245 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:47:53.162503 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0619 16:47:53.164075 (+ 1572us) server_negotiation.cc:299] Negotiation successful
0619 16:47:53.164234 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15797,"thread_start_us":15662,"threads_started":1}
I20260619 16:47:53.166136 18207 tablet_service.cc:1467] Processing CreateTablet for tablet e6aa1bab084d40f2bc15aedacbfaf0fe (DEFAULT_TABLE table=test-table [id=2f301654382b4bc69d81dce5f161a870]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:47:53.166713 18338 tablet_service.cc:1467] Processing CreateTablet for tablet e6aa1bab084d40f2bc15aedacbfaf0fe (DEFAULT_TABLE table=test-table [id=2f301654382b4bc69d81dce5f161a870]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:47:53.167411 18207 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e6aa1bab084d40f2bc15aedacbfaf0fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:47:53.168524 18338 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e6aa1bab084d40f2bc15aedacbfaf0fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:47:53.167970 18468 tablet_service.cc:1467] Processing CreateTablet for tablet e6aa1bab084d40f2bc15aedacbfaf0fe (DEFAULT_TABLE table=test-table [id=2f301654382b4bc69d81dce5f161a870]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:47:53.169358 18468 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e6aa1bab084d40f2bc15aedacbfaf0fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:47:53.178166 18551 tablet_bootstrap.cc:492] T e6aa1bab084d40f2bc15aedacbfaf0fe P 61cdcb94fbae4942b904f01024e847ee: Bootstrap starting.
I20260619 16:47:53.181106 18551 tablet_bootstrap.cc:654] T e6aa1bab084d40f2bc15aedacbfaf0fe P 61cdcb94fbae4942b904f01024e847ee: Neither blocks nor log segments found. Creating new log.
I20260619 16:47:53.182209 18551 log.cc:824] T e6aa1bab084d40f2bc15aedacbfaf0fe P 61cdcb94fbae4942b904f01024e847ee: Log is configured to *not* fsync() on all Append() calls
I20260619 16:47:53.182304 18552 tablet_bootstrap.cc:492] T e6aa1bab084d40f2bc15aedacbfaf0fe P d186a9b6f13c4bcca28612936da1f89d: Bootstrap starting.
I20260619 16:47:53.183492 18553 tablet_bootstrap.cc:492] T e6aa1bab084d40f2bc15aedacbfaf0fe P 73d867d2d584466fa4d8c22fabcdfccb: Bootstrap starting.
I20260619 16:47:53.185009 18551 tablet_bootstrap.cc:492] T e6aa1bab084d40f2bc15aedacbfaf0fe P 61cdcb94fbae4942b904f01024e847ee: No bootstrap required, opened a new log
I20260619 16:47:53.185242 18551 ts_tablet_manager.cc:1397] T e6aa1bab084d40f2bc15aedacbfaf0fe P 61cdcb94fbae4942b904f01024e847ee: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.002s
I20260619 16:47:53.186049 18552 tablet_bootstrap.cc:654] T e6aa1bab084d40f2bc15aedacbfaf0fe P d186a9b6f13c4bcca28612936da1f89d: Neither blocks nor log segments found. Creating new log.
I20260619 16:47:53.186717 18553 tablet_bootstrap.cc:654] T e6aa1bab084d40f2bc15aedacbfaf0fe P 73d867d2d584466fa4d8c22fabcdfccb: Neither blocks nor log segments found. Creating new log.
I20260619 16:47:53.187216 18552 log.cc:824] T e6aa1bab084d40f2bc15aedacbfaf0fe P d186a9b6f13c4bcca28612936da1f89d: Log is configured to *not* fsync() on all Append() calls
I20260619 16:47:53.187801 18553 log.cc:824] T e6aa1bab084d40f2bc15aedacbfaf0fe P 73d867d2d584466fa4d8c22fabcdfccb: Log is configured to *not* fsync() on all Append() calls
I20260619 16:47:53.190097 18552 tablet_bootstrap.cc:492] T e6aa1bab084d40f2bc15aedacbfaf0fe P d186a9b6f13c4bcca28612936da1f89d: No bootstrap required, opened a new log
I20260619 16:47:53.190351 18552 ts_tablet_manager.cc:1397] T e6aa1bab084d40f2bc15aedacbfaf0fe P d186a9b6f13c4bcca28612936da1f89d: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.002s
I20260619 16:47:53.192284 18553 tablet_bootstrap.cc:492] T e6aa1bab084d40f2bc15aedacbfaf0fe P 73d867d2d584466fa4d8c22fabcdfccb: No bootstrap required, opened a new log
I20260619 16:47:53.192544 18553 ts_tablet_manager.cc:1397] T e6aa1bab084d40f2bc15aedacbfaf0fe P 73d867d2d584466fa4d8c22fabcdfccb: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.007s
I20260619 16:47:53.193781 18551 raft_consensus.cc:348] T e6aa1bab084d40f2bc15aedacbfaf0fe P 61cdcb94fbae4942b904f01024e847ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61cdcb94fbae4942b904f01024e847ee" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45595 } } peers { permanent_uuid: "73d867d2d584466fa4d8c22fabcdfccb" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42709 } } peers { permanent_uuid: "d186a9b6f13c4bcca28612936da1f89d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42429 } }
I20260619 16:47:53.194238 18551 raft_consensus.cc:374] T e6aa1bab084d40f2bc15aedacbfaf0fe P 61cdcb94fbae4942b904f01024e847ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:47:53.194324 18551 raft_consensus.cc:729] T e6aa1bab084d40f2bc15aedacbfaf0fe P 61cdcb94fbae4942b904f01024e847ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61cdcb94fbae4942b904f01024e847ee, State: Initialized, Role: FOLLOWER
I20260619 16:47:53.195070 18551 consensus_queue.cc:260] T e6aa1bab084d40f2bc15aedacbfaf0fe P 61cdcb94fbae4942b904f01024e847ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61cdcb94fbae4942b904f01024e847ee" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45595 } } peers { permanent_uuid: "73d867d2d584466fa4d8c22fabcdfccb" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42709 } } peers { permanent_uuid: "d186a9b6f13c4bcca28612936da1f89d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42429 } }
I20260619 16:47:53.196130 18272 heartbeater.cc:503] Master 127.0.97.62:46581 was elected leader, sending a full tablet report...
I20260619 16:47:53.196751 18551 ts_tablet_manager.cc:1428] T e6aa1bab084d40f2bc15aedacbfaf0fe P 61cdcb94fbae4942b904f01024e847ee: Time spent starting tablet: real 0.011s user 0.011s sys 0.001s
I20260619 16:47:53.198235 18552 raft_consensus.cc:348] T e6aa1bab084d40f2bc15aedacbfaf0fe P d186a9b6f13c4bcca28612936da1f89d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61cdcb94fbae4942b904f01024e847ee" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45595 } } peers { permanent_uuid: "73d867d2d584466fa4d8c22fabcdfccb" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42709 } } peers { permanent_uuid: "d186a9b6f13c4bcca28612936da1f89d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42429 } }
I20260619 16:47:53.198546 18552 raft_consensus.cc:374] T e6aa1bab084d40f2bc15aedacbfaf0fe P d186a9b6f13c4bcca28612936da1f89d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:47:53.198655 18552 raft_consensus.cc:729] T e6aa1bab084d40f2bc15aedacbfaf0fe P d186a9b6f13c4bcca28612936da1f89d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d186a9b6f13c4bcca28612936da1f89d, State: Initialized, Role: FOLLOWER
I20260619 16:47:53.199172 18552 consensus_queue.cc:260] T e6aa1bab084d40f2bc15aedacbfaf0fe P d186a9b6f13c4bcca28612936da1f89d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61cdcb94fbae4942b904f01024e847ee" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45595 } } peers { permanent_uuid: "73d867d2d584466fa4d8c22fabcdfccb" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42709 } } peers { permanent_uuid: "d186a9b6f13c4bcca28612936da1f89d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42429 } }
I20260619 16:47:53.201510 18553 raft_consensus.cc:348] T e6aa1bab084d40f2bc15aedacbfaf0fe P 73d867d2d584466fa4d8c22fabcdfccb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61cdcb94fbae4942b904f01024e847ee" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45595 } } peers { permanent_uuid: "73d867d2d584466fa4d8c22fabcdfccb" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42709 } } peers { permanent_uuid: "d186a9b6f13c4bcca28612936da1f89d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42429 } }
I20260619 16:47:53.201907 18553 raft_consensus.cc:374] T e6aa1bab084d40f2bc15aedacbfaf0fe P 73d867d2d584466fa4d8c22fabcdfccb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:47:53.202015 18553 raft_consensus.cc:729] T e6aa1bab084d40f2bc15aedacbfaf0fe P 73d867d2d584466fa4d8c22fabcdfccb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73d867d2d584466fa4d8c22fabcdfccb, State: Initialized, Role: FOLLOWER
I20260619 16:47:53.202508 18403 heartbeater.cc:503] Master 127.0.97.62:46581 was elected leader, sending a full tablet report...
I20260619 16:47:53.202610 18553 consensus_queue.cc:260] T e6aa1bab084d40f2bc15aedacbfaf0fe P 73d867d2d584466fa4d8c22fabcdfccb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61cdcb94fbae4942b904f01024e847ee" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45595 } } peers { permanent_uuid: "73d867d2d584466fa4d8c22fabcdfccb" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42709 } } peers { permanent_uuid: "d186a9b6f13c4bcca28612936da1f89d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42429 } }
I20260619 16:47:53.203225 18552 ts_tablet_manager.cc:1428] T e6aa1bab084d40f2bc15aedacbfaf0fe P d186a9b6f13c4bcca28612936da1f89d: Time spent starting tablet: real 0.013s user 0.002s sys 0.010s
I20260619 16:47:53.203583 18533 heartbeater.cc:503] Master 127.0.97.62:46581 was elected leader, sending a full tablet report...
I20260619 16:47:53.204165 18553 ts_tablet_manager.cc:1428] T e6aa1bab084d40f2bc15aedacbfaf0fe P 73d867d2d584466fa4d8c22fabcdfccb: Time spent starting tablet: real 0.012s user 0.011s sys 0.001s
I20260619 16:47:53.236285 18557 raft_consensus.cc:482] T e6aa1bab084d40f2bc15aedacbfaf0fe P 61cdcb94fbae4942b904f01024e847ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 16:47:53.236575 18557 raft_consensus.cc:504] T e6aa1bab084d40f2bc15aedacbfaf0fe P 61cdcb94fbae4942b904f01024e847ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61cdcb94fbae4942b904f01024e847ee" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45595 } } peers { permanent_uuid: "73d867d2d584466fa4d8c22fabcdfccb" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42709 } } peers { permanent_uuid: "d186a9b6f13c4bcca28612936da1f89d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42429 } }
I20260619 16:47:53.238337 18557 leader_election.cc:288] T e6aa1bab084d40f2bc15aedacbfaf0fe P 61cdcb94fbae4942b904f01024e847ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 73d867d2d584466fa4d8c22fabcdfccb (127.0.97.3:42709), d186a9b6f13c4bcca28612936da1f89d (127.0.97.2:42429)
W20260619 16:47:53.242977 18273 tablet.cc:2369] T e6aa1bab084d40f2bc15aedacbfaf0fe P 61cdcb94fbae4942b904f01024e847ee: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 16:47:53.246606 18550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:47:53.238828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:55809 (local address 127.0.97.2:42429)
0619 16:47:53.239030 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0619 16:47:53.239037 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:47:53.239516 (+ 479us) server_negotiation.cc:407] Connection header received
0619 16:47:53.239795 (+ 279us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:47:53.239800 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:47:53.239874 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:47:53.239970 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:47:53.240873 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:47:53.241897 (+ 1024us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:47:53.244969 (+ 3072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:47:53.245782 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:47:53.245897 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:47:53.245991 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0619 16:47:53.246251 (+ 260us) server_negotiation.cc:299] Negotiation successful
0619 16:47:53.246369 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260619 16:47:53.246724 18560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:47:53.238609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:42429 (local address 127.0.97.1:55809)
0619 16:47:53.239406 (+ 797us) negotiation.cc:107] Waiting for socket to connect
0619 16:47:53.239426 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 16:47:53.239638 (+ 212us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:47:53.240026 (+ 388us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:47:53.240030 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:47:53.240094 (+ 64us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:47:53.240690 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:47:53.240702 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:47:53.242055 (+ 1353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:47:53.242059 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:47:53.244677 (+ 2618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:47:53.244691 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:47:53.245953 (+ 1262us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:47:53.245959 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:47:53.246113 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:47:53.246136 (+ 23us) client_negotiation.cc:769] Sending connection context
0619 16:47:53.246213 (+ 77us) client_negotiation.cc:240] Negotiation successful
0619 16:47:53.246331 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":613,"mutex_wait_us":135,"spinlock_wait_cycles":20224,"thread_start_us":447,"threads_started":1}
I20260619 16:47:53.247184 18548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:47:53.238658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:48163 (local address 127.0.97.3:42709)
0619 16:47:53.238861 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0619 16:47:53.238867 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:47:53.239812 (+ 945us) server_negotiation.cc:407] Connection header received
0619 16:47:53.239964 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:47:53.239969 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:47:53.240037 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:47:53.240112 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:47:53.240770 (+ 658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:47:53.241927 (+ 1157us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:47:53.244859 (+ 2932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:47:53.246145 (+ 1286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:47:53.246253 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:47:53.246344 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0619 16:47:53.246824 (+ 480us) server_negotiation.cc:299] Negotiation successful
0619 16:47:53.246943 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260619 16:47:53.247577 18358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6aa1bab084d40f2bc15aedacbfaf0fe" candidate_uuid: "61cdcb94fbae4942b904f01024e847ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d186a9b6f13c4bcca28612936da1f89d" is_pre_election: true
I20260619 16:47:53.247776 18561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:47:53.238659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:42709 (local address 127.0.97.1:48163)
0619 16:47:53.239704 (+ 1045us) negotiation.cc:107] Waiting for socket to connect
0619 16:47:53.239721 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 16:47:53.239852 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:47:53.240155 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:47:53.240160 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:47:53.240196 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:47:53.240607 (+ 411us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:47:53.240617 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:47:53.242052 (+ 1435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:47:53.242057 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:47:53.244678 (+ 2621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:47:53.244691 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:47:53.246459 (+ 1768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:47:53.246465 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:47:53.246649 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:47:53.246682 (+ 33us) client_negotiation.cc:769] Sending connection context
0619 16:47:53.247341 (+ 659us) client_negotiation.cc:240] Negotiation successful
0619 16:47:53.247464 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":919,"mutex_wait_us":191,"thread_start_us":722,"threads_started":1}
I20260619 16:47:53.248034 18358 raft_consensus.cc:2454] T e6aa1bab084d40f2bc15aedacbfaf0fe P d186a9b6f13c4bcca28612936da1f89d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 61cdcb94fbae4942b904f01024e847ee in term 0.
I20260619 16:47:53.248670 18162 leader_election.cc:302] T e6aa1bab084d40f2bc15aedacbfaf0fe P 61cdcb94fbae4942b904f01024e847ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61cdcb94fbae4942b904f01024e847ee, d186a9b6f13c4bcca28612936da1f89d; no voters:
I20260619 16:47:53.249090 18557 raft_consensus.cc:2781] T e6aa1bab084d40f2bc15aedacbfaf0fe P 61cdcb94fbae4942b904f01024e847ee [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 16:47:53.249300 18557 raft_consensus.cc:482] T e6aa1bab084d40f2bc15aedacbfaf0fe P 61cdcb94fbae4942b904f01024e847ee [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 16:47:53.249403 18557 raft_consensus.cc:3037] T e6aa1bab084d40f2bc15aedacbfaf0fe P 61cdcb94fbae4942b904f01024e847ee [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:47:53.249210 18488 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6aa1bab084d40f2bc15aedacbfaf0fe" candidate_uuid: "61cdcb94fbae4942b904f01024e847ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73d867d2d584466fa4d8c22fabcdfccb" is_pre_election: true
I20260619 16:47:53.249715 18488 raft_consensus.cc:2454] T e6aa1bab084d40f2bc15aedacbfaf0fe P 73d867d2d584466fa4d8c22fabcdfccb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 61cdcb94fbae4942b904f01024e847ee in term 0.
I20260619 16:47:53.251950 18557 raft_consensus.cc:504] T e6aa1bab084d40f2bc15aedacbfaf0fe P 61cdcb94fbae4942b904f01024e847ee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61cdcb94fbae4942b904f01024e847ee" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45595 } } peers { permanent_uuid: "73d867d2d584466fa4d8c22fabcdfccb" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42709 } } peers { permanent_uuid: "d186a9b6f13c4bcca28612936da1f89d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42429 } }
I20260619 16:47:53.252514 18557 leader_election.cc:288] T e6aa1bab084d40f2bc15aedacbfaf0fe P 61cdcb94fbae4942b904f01024e847ee [CANDIDATE]: Term 1 election: Requested vote from peers 73d867d2d584466fa4d8c22fabcdfccb (127.0.97.3:42709), d186a9b6f13c4bcca28612936da1f89d (127.0.97.2:42429)
I20260619 16:47:53.253085 18488 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6aa1bab084d40f2bc15aedacbfaf0fe" candidate_uuid: "61cdcb94fbae4942b904f01024e847ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73d867d2d584466fa4d8c22fabcdfccb"
I20260619 16:47:53.253121 18358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6aa1bab084d40f2bc15aedacbfaf0fe" candidate_uuid: "61cdcb94fbae4942b904f01024e847ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d186a9b6f13c4bcca28612936da1f89d"
I20260619 16:47:53.253242 18488 raft_consensus.cc:3037] T e6aa1bab084d40f2bc15aedacbfaf0fe P 73d867d2d584466fa4d8c22fabcdfccb [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:47:53.253291 18358 raft_consensus.cc:3037] T e6aa1bab084d40f2bc15aedacbfaf0fe P d186a9b6f13c4bcca28612936da1f89d [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:47:53.255412 18488 raft_consensus.cc:2454] T e6aa1bab084d40f2bc15aedacbfaf0fe P 73d867d2d584466fa4d8c22fabcdfccb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 61cdcb94fbae4942b904f01024e847ee in term 1.
I20260619 16:47:53.255540 18358 raft_consensus.cc:2454] T e6aa1bab084d40f2bc15aedacbfaf0fe P d186a9b6f13c4bcca28612936da1f89d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 61cdcb94fbae4942b904f01024e847ee in term 1.
I20260619 16:47:53.255766 18163 leader_election.cc:302] T e6aa1bab084d40f2bc15aedacbfaf0fe P 61cdcb94fbae4942b904f01024e847ee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61cdcb94fbae4942b904f01024e847ee, 73d867d2d584466fa4d8c22fabcdfccb; no voters:
I20260619 16:47:53.256081 18557 raft_consensus.cc:2781] T e6aa1bab084d40f2bc15aedacbfaf0fe P 61cdcb94fbae4942b904f01024e847ee [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:47:53.256490 18557 raft_consensus.cc:686] T e6aa1bab084d40f2bc15aedacbfaf0fe P 61cdcb94fbae4942b904f01024e847ee [term 1 LEADER]: Becoming Leader. State: Replica: 61cdcb94fbae4942b904f01024e847ee, State: Running, Role: LEADER
I20260619 16:47:53.256911 18557 consensus_queue.cc:237] T e6aa1bab084d40f2bc15aedacbfaf0fe P 61cdcb94fbae4942b904f01024e847ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61cdcb94fbae4942b904f01024e847ee" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45595 } } peers { permanent_uuid: "73d867d2d584466fa4d8c22fabcdfccb" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42709 } } peers { permanent_uuid: "d186a9b6f13c4bcca28612936da1f89d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42429 } }
I20260619 16:47:53.261801 18085 catalog_manager.cc:5495] T e6aa1bab084d40f2bc15aedacbfaf0fe P 61cdcb94fbae4942b904f01024e847ee reported cstate change: term changed from 0 to 1, leader changed from <none> to 61cdcb94fbae4942b904f01024e847ee (127.0.97.1). New cstate: current_term: 1 leader_uuid: "61cdcb94fbae4942b904f01024e847ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61cdcb94fbae4942b904f01024e847ee" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45595 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "73d867d2d584466fa4d8c22fabcdfccb" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42709 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d186a9b6f13c4bcca28612936da1f89d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42429 } health_report { overall_health: UNKNOWN } } }
I20260619 16:47:53.301761 18546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:47:53.296455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34772 (local address 127.0.97.1:45595)
0619 16:47:53.296702 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0619 16:47:53.296708 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:47:53.296727 (+ 19us) server_negotiation.cc:407] Connection header received
0619 16:47:53.296954 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:47:53.296958 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:47:53.297045 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:47:53.297177 (+ 132us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 16:47:53.297536 (+ 359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:47:53.298266 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:47:53.299633 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:47:53.300023 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:47:53.300134 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:47:53.300626 (+ 492us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 16:47:53.301177 (+ 551us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 16:47:53.301305 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0619 16:47:53.301399 (+ 94us) server_negotiation.cc:299] Negotiation successful
0619 16:47:53.301474 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260619 16:47:53.309453 18358 raft_consensus.cc:1261] T e6aa1bab084d40f2bc15aedacbfaf0fe P d186a9b6f13c4bcca28612936da1f89d [term 1 FOLLOWER]: Refusing update from remote peer 61cdcb94fbae4942b904f01024e847ee: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:47:53.309459 18488 raft_consensus.cc:1261] T e6aa1bab084d40f2bc15aedacbfaf0fe P 73d867d2d584466fa4d8c22fabcdfccb [term 1 FOLLOWER]: Refusing update from remote peer 61cdcb94fbae4942b904f01024e847ee: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:47:53.310376 18562 consensus_queue.cc:1035] T e6aa1bab084d40f2bc15aedacbfaf0fe P 61cdcb94fbae4942b904f01024e847ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "d186a9b6f13c4bcca28612936da1f89d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42429 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:47:53.310708 18557 consensus_queue.cc:1035] T e6aa1bab084d40f2bc15aedacbfaf0fe P 61cdcb94fbae4942b904f01024e847ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "73d867d2d584466fa4d8c22fabcdfccb" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42709 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:47:53.325525 18567 mvcc.cc:205] Tried to move back new op lower bound from 7298611909866467328 to 7298611909664022528. Current Snapshot: MvccSnapshot[applied={T|T < 7298611909866467328}]
I20260619 16:47:53.333395 18086 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:60778:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:47:53.333686 18086 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:60778:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:47:53.337010 18086 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 003e2cf86ec44b63be7819e7085826e6: Sending DeleteTablet for 3 replicas of tablet e6aa1bab084d40f2bc15aedacbfaf0fe
I20260619 16:47:53.338213 18468 tablet_service.cc:1514] Processing DeleteTablet for tablet e6aa1bab084d40f2bc15aedacbfaf0fe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:47:53 UTC) from {username='slave'} at 127.0.0.1:34392
I20260619 16:47:53.338224 18338 tablet_service.cc:1514] Processing DeleteTablet for tablet e6aa1bab084d40f2bc15aedacbfaf0fe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:47:53 UTC) from {username='slave'} at 127.0.0.1:49986
I20260619 16:47:53.339164 18207 tablet_service.cc:1514] Processing DeleteTablet for tablet e6aa1bab084d40f2bc15aedacbfaf0fe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:47:53 UTC) from {username='slave'} at 127.0.0.1:34756
I20260619 16:47:53.339536 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu pid 18145
I20260619 16:47:53.352622 18577 tablet_replica.cc:330] T e6aa1bab084d40f2bc15aedacbfaf0fe P 61cdcb94fbae4942b904f01024e847ee: stopping tablet replica
I20260619 16:47:53.353143 18577 raft_consensus.cc:2229] T e6aa1bab084d40f2bc15aedacbfaf0fe P 61cdcb94fbae4942b904f01024e847ee [term 1 LEADER]: Raft consensus shutting down.
I20260619 16:47:53.353868 18577 raft_consensus.cc:2258] T e6aa1bab084d40f2bc15aedacbfaf0fe P 61cdcb94fbae4942b904f01024e847ee [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:47:53.356508 18576 tablet_replica.cc:330] T e6aa1bab084d40f2bc15aedacbfaf0fe P d186a9b6f13c4bcca28612936da1f89d: stopping tablet replica
I20260619 16:47:53.357084 18576 raft_consensus.cc:2229] T e6aa1bab084d40f2bc15aedacbfaf0fe P d186a9b6f13c4bcca28612936da1f89d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 16:47:53.358229 18575 tablet_replica.cc:330] T e6aa1bab084d40f2bc15aedacbfaf0fe P 73d867d2d584466fa4d8c22fabcdfccb: stopping tablet replica
I20260619 16:47:53.357683 18576 raft_consensus.cc:2258] T e6aa1bab084d40f2bc15aedacbfaf0fe P d186a9b6f13c4bcca28612936da1f89d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:47:53.359622 18575 raft_consensus.cc:2229] T e6aa1bab084d40f2bc15aedacbfaf0fe P 73d867d2d584466fa4d8c22fabcdfccb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 16:47:53.360116 18575 raft_consensus.cc:2258] T e6aa1bab084d40f2bc15aedacbfaf0fe P 73d867d2d584466fa4d8c22fabcdfccb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:47:53.360330 18577 ts_tablet_manager.cc:1905] T e6aa1bab084d40f2bc15aedacbfaf0fe P 61cdcb94fbae4942b904f01024e847ee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:47:53.360625 18576 ts_tablet_manager.cc:1905] T e6aa1bab084d40f2bc15aedacbfaf0fe P d186a9b6f13c4bcca28612936da1f89d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:47:53.362464 18546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:47:53.341941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34788 (local address 127.0.97.1:45595)
0619 16:47:53.342384 (+ 443us) server_negotiation.cc:206] Beginning negotiation
0619 16:47:53.342390 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:47:53.350798 (+ 8408us) server_negotiation.cc:407] Connection header received
0619 16:47:53.351232 (+ 434us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:47:53.351238 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:47:53.351341 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:47:53.351423 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:47:53.354306 (+ 2883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:47:53.355781 (+ 1475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:47:53.357721 (+ 1940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:47:53.358385 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:47:53.358487 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:47:53.360756 (+ 2269us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:47:53.360766 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:47:53.360784 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:47:53.361537 (+ 753us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:47:53.361596 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:47:53.361622 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:47:53.361696 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0619 16:47:53.361955 (+ 259us) server_negotiation.cc:299] Negotiation successful
0619 16:47:53.362069 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":322}
I20260619 16:47:53.363493 18267 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:47:53.366274 18575 ts_tablet_manager.cc:1905] T e6aa1bab084d40f2bc15aedacbfaf0fe P 73d867d2d584466fa4d8c22fabcdfccb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:47:53.368780 18576 ts_tablet_manager.cc:1918] T e6aa1bab084d40f2bc15aedacbfaf0fe P d186a9b6f13c4bcca28612936da1f89d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 16:47:53.368934 18576 log.cc:1196] T e6aa1bab084d40f2bc15aedacbfaf0fe P d186a9b6f13c4bcca28612936da1f89d: Deleting WAL directory at /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781887385903558-388-0/minicluster-data/ts-1/wal/wals/e6aa1bab084d40f2bc15aedacbfaf0fe
I20260619 16:47:53.369405 18576 ts_tablet_manager.cc:1939] T e6aa1bab084d40f2bc15aedacbfaf0fe P d186a9b6f13c4bcca28612936da1f89d: Deleting consensus metadata
I20260619 16:47:53.371243 18072 catalog_manager.cc:4841] TS d186a9b6f13c4bcca28612936da1f89d (127.0.97.2:42429): tablet e6aa1bab084d40f2bc15aedacbfaf0fe (table test-table [id=2f301654382b4bc69d81dce5f161a870]) successfully deleted
I20260619 16:47:53.371932 18575 ts_tablet_manager.cc:1918] T e6aa1bab084d40f2bc15aedacbfaf0fe P 73d867d2d584466fa4d8c22fabcdfccb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 16:47:53.372069 18575 log.cc:1196] T e6aa1bab084d40f2bc15aedacbfaf0fe P 73d867d2d584466fa4d8c22fabcdfccb: Deleting WAL directory at /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781887385903558-388-0/minicluster-data/ts-2/wal/wals/e6aa1bab084d40f2bc15aedacbfaf0fe
I20260619 16:47:53.372421 18575 ts_tablet_manager.cc:1939] T e6aa1bab084d40f2bc15aedacbfaf0fe P 73d867d2d584466fa4d8c22fabcdfccb: Deleting consensus metadata
I20260619 16:47:53.373795 18073 catalog_manager.cc:4841] TS 73d867d2d584466fa4d8c22fabcdfccb (127.0.97.3:42709): tablet e6aa1bab084d40f2bc15aedacbfaf0fe (table test-table [id=2f301654382b4bc69d81dce5f161a870]) successfully deleted
I20260619 16:47:53.460352 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu with pid 18145
I20260619 16:47:53.474555 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu pid 18276
W20260619 16:47:53.474633 18071 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.97.1:45595 (error 108)
I20260619 16:47:53.476431 18545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:47:53.476020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45595 (local address 127.0.0.1:34800)
0619 16:47:53.476180 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0619 16:47:53.476278 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:45595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260619 16:47:53.476709 18071 catalog_manager.cc:4584] TS 61cdcb94fbae4942b904f01024e847ee (127.0.97.1:45595): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e6aa1bab084d40f2bc15aedacbfaf0fe: Network error: Client connection negotiation failed: client connection to 127.0.97.1:45595: connect: Connection refused (error 111)
I20260619 16:47:53.481132 18550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:47:53.475351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49998 (local address 127.0.97.2:42429)
0619 16:47:53.475542 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0619 16:47:53.475549 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:47:53.475590 (+ 41us) server_negotiation.cc:407] Connection header received
0619 16:47:53.475762 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:47:53.475768 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:47:53.475872 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:47:53.475969 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:47:53.476399 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:47:53.477256 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:47:53.478508 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:47:53.478918 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:47:53.479016 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:47:53.479907 (+ 891us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:47:53.479915 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:47:53.479933 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:47:53.480503 (+ 570us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:47:53.480553 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:47:53.480573 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:47:53.480660 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0619 16:47:53.480791 (+ 131us) server_negotiation.cc:299] Negotiation successful
0619 16:47:53.480882 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260619 16:47:53.481961 18398 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:47:53.530328 18545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:47:53.529791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45595 (local address 127.0.0.1:34808)
0619 16:47:53.530017 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0619 16:47:53.530130 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:45595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260619 16:47:53.531445 18545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:47:53.531057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45595 (local address 127.0.0.1:34820)
0619 16:47:53.531207 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0619 16:47:53.531302 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:45595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260619 16:47:53.569087 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu with pid 18276
I20260619 16:47:53.582202 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu pid 18406
I20260619 16:47:53.592008 18545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:47:53.591551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45595 (local address 127.0.0.1:34828)
0619 16:47:53.591769 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0619 16:47:53.591850 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:45595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260619 16:47:53.592494 18548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:47:53.586461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34408 (local address 127.0.97.3:42709)
0619 16:47:53.586615 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0619 16:47:53.586620 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:47:53.586658 (+ 38us) server_negotiation.cc:407] Connection header received
0619 16:47:53.586825 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:47:53.586828 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:47:53.586909 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:47:53.586993 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:47:53.587359 (+ 366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:47:53.588099 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:47:53.589436 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:47:53.589888 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:47:53.589995 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:47:53.590958 (+ 963us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:47:53.590967 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:47:53.590982 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:47:53.591777 (+ 795us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:47:53.591825 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:47:53.591846 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:47:53.591941 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0619 16:47:53.592077 (+ 136us) server_negotiation.cc:299] Negotiation successful
0619 16:47:53.592180 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260619 16:47:53.593101 18528 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:47:53.593122 18545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:47:53.592748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45595 (local address 127.0.0.1:34832)
0619 16:47:53.592917 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0619 16:47:53.592996 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:45595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260619 16:47:53.673805 18545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:47:53.673288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45595 (local address 127.0.0.1:34842)
0619 16:47:53.673496 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0619 16:47:53.673589 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:45595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260619 16:47:53.674813 18545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:47:53.674419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45595 (local address 127.0.0.1:34848)
0619 16:47:53.674610 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0619 16:47:53.674705 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:45595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
I20260619 16:47:53.679713 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu with pid 18406
I20260619 16:47:53.692958 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu pid 18055
I20260619 16:47:53.693982 18116 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:47:53.800488 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu with pid 18055
2026-06-19T16:47:53Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (2568 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887385903558-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 16:47:53 dist-test-slave-wvz7 krb5kdc[18586](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 16:47:53 dist-test-slave-wvz7 krb5kdc[18586](info): set up 2 sockets
Jun 19 16:47:53 dist-test-slave-wvz7 krb5kdc[18586](info): commencing operation
krb5kdc: starting...
W20260619 16:47:54.809474 388 mini_kdc.cc:121] Time spent starting KDC: real 0.973s user 0.000s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887385903558-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887385903558-388-0/krb5kdc/test-user.keytab.
Jun 19 16:47:54 dist-test-slave-wvz7 krb5kdc[18586](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887674, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T16:47:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T16:47:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887385903558-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887385903558-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260619 16:47:55.034677 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu
/tmp/dist-test-taskEs2EH8/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887385903558-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887385903558-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887385903558-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887385903558-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34357
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887385903558-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38481
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:34357
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887385903558-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887385903558-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887385903558-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:47:55.366350 18599 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:47:55.366647 18599 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:47:55.366761 18599 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:47:55.374369 18599 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:47:55.374495 18599 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:47:55.374544 18599 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:47:55.374620 18599 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:47:55.374676 18599 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:47:55.384464 18599 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38481
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887385903558-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887385903558-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:34357
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887385903558-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34357
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887385903558-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887385903558-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:47:55.385721 18599 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:47:55.387025 18599 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:47:55.398185 18605 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:47:55.398244 18604 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:47:55.398223 18607 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:47:55.398919 18599 server_base.cc:1034] running on GCE node
I20260619 16:47:55.399628 18599 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:47:55.401039 18599 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:47:55.402233 18599 hybrid_clock.cc:648] HybridClock initialized: now 1781887675402193 us; error 28 us; skew 500 ppm
I20260619 16:47:55.402599 18599 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:47:55 dist-test-slave-wvz7 krb5kdc[18586](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887675, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:47:55.405766 18599 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260619 16:47:55.407198 18599 webserver.cc:460] Webserver started at http://127.0.97.62:45861/ using document root <none> and password file <none>
I20260619 16:47:55.407835 18599 fs_manager.cc:359] Metadata directory not provided
I20260619 16:47:55.407929 18599 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:47:55.408178 18599 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:47:55.411020 18599 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887385903558-388-0/minicluster-data/master-0/data/instance:
uuid: "c30384d62f134e9f841056be2f40b0f3"
format_stamp: "Formatted at 2026-06-19 16:47:55 on dist-test-slave-wvz7"
server_key: "84606de979785e61f62cfb028c691836"
server_key_iv: "994ab0658363b5883ad4ebf019af1fa3"
server_key_version: "encryptionkey@0"
I20260619 16:47:55.411706 18599 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887385903558-388-0/minicluster-data/master-0/wal/instance:
uuid: "c30384d62f134e9f841056be2f40b0f3"
format_stamp: "Formatted at 2026-06-19 16:47:55 on dist-test-slave-wvz7"
server_key: "84606de979785e61f62cfb028c691836"
server_key_iv: "994ab0658363b5883ad4ebf019af1fa3"
server_key_version: "encryptionkey@0"
I20260619 16:47:55.416558 18599 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 16:47:55.419708 18614 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:47:55.420910 18599 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 16:47:55.421044 18599 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887385903558-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887385903558-388-0/minicluster-data/master-0/wal
uuid: "c30384d62f134e9f841056be2f40b0f3"
format_stamp: "Formatted at 2026-06-19 16:47:55 on dist-test-slave-wvz7"
server_key: "84606de979785e61f62cfb028c691836"
server_key_iv: "994ab0658363b5883ad4ebf019af1fa3"
server_key_version: "encryptionkey@0"
I20260619 16:47:55.421211 18599 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887385903558-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887385903558-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887385903558-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:47:55.460364 18599 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:47:55.464545 18599 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:47:55.464865 18599 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:47:55.483187 18599 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:34357
I20260619 16:47:55.483228 18665 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:34357 every 8 connection(s)
I20260619 16:47:55.484679 18599 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887385903558-388-0/minicluster-data/master-0/data/info.pb
I20260619 16:47:55.487659 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu as pid 18599
I20260619 16:47:55.487829 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887385903558-388-0/minicluster-data/master-0/wal/instance
I20260619 16:47:55.488260 388 external_mini_cluster.cc:1428] Setting key ae4a47c35352744bdc06d128a643321c
I20260619 16:47:55.488583 18666 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:47:55.502458 18666 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c30384d62f134e9f841056be2f40b0f3: Bootstrap starting.
Jun 19 16:47:55 dist-test-slave-wvz7 krb5kdc[18586](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887674, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 16:47:55.506042 18666 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c30384d62f134e9f841056be2f40b0f3: Neither blocks nor log segments found. Creating new log.
I20260619 16:47:55.507357 18666 log.cc:824] T 00000000000000000000000000000000 P c30384d62f134e9f841056be2f40b0f3: Log is configured to *not* fsync() on all Append() calls
I20260619 16:47:55.510629 18666 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c30384d62f134e9f841056be2f40b0f3: No bootstrap required, opened a new log
I20260619 16:47:55.512336 18669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:47:55.490221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49222 (local address 127.0.97.62:34357)
0619 16:47:55.490979 (+ 758us) server_negotiation.cc:206] Beginning negotiation
0619 16:47:55.490994 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 16:47:55.491046 (+ 52us) server_negotiation.cc:407] Connection header received
0619 16:47:55.493324 (+ 2278us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:47:55.493352 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:47:55.493910 (+ 558us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:47:55.494384 (+ 474us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:47:55.495643 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:47:55.497594 (+ 1951us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:47:55.498919 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:47:55.499979 (+ 1060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:47:55.500180 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:47:55.504086 (+ 3906us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:47:55.504099 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:47:55.504119 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:47:55.504165 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:47:55.507874 (+ 3709us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:47:55.508708 (+ 834us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:47:55.508717 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:47:55.508725 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:47:55.508931 (+ 206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:47:55.509290 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:47:55.509295 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:47:55.509298 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:47:55.509984 (+ 686us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:47:55.510270 (+ 286us) server_negotiation.cc:1035] Waiting for connection context
0619 16:47:55.510631 (+ 361us) server_negotiation.cc:299] Negotiation successful
0619 16:47:55.511060 (+ 429us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":525,"thread_start_us":300,"threads_started":1}
I20260619 16:47:55.518988 18666 raft_consensus.cc:348] T 00000000000000000000000000000000 P c30384d62f134e9f841056be2f40b0f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c30384d62f134e9f841056be2f40b0f3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34357 } }
I20260619 16:47:55.519389 18666 raft_consensus.cc:374] T 00000000000000000000000000000000 P c30384d62f134e9f841056be2f40b0f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:47:55.519497 18666 raft_consensus.cc:729] T 00000000000000000000000000000000 P c30384d62f134e9f841056be2f40b0f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c30384d62f134e9f841056be2f40b0f3, State: Initialized, Role: FOLLOWER
I20260619 16:47:55.520020 18666 consensus_queue.cc:260] T 00000000000000000000000000000000 P c30384d62f134e9f841056be2f40b0f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c30384d62f134e9f841056be2f40b0f3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34357 } }
I20260619 16:47:55.520222 18666 raft_consensus.cc:388] T 00000000000000000000000000000000 P c30384d62f134e9f841056be2f40b0f3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:47:55.520337 18666 raft_consensus.cc:482] T 00000000000000000000000000000000 P c30384d62f134e9f841056be2f40b0f3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:47:55.520476 18666 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c30384d62f134e9f841056be2f40b0f3 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:47:55.522552 18666 raft_consensus.cc:504] T 00000000000000000000000000000000 P c30384d62f134e9f841056be2f40b0f3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c30384d62f134e9f841056be2f40b0f3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34357 } }
I20260619 16:47:55.523012 18666 leader_election.cc:302] T 00000000000000000000000000000000 P c30384d62f134e9f841056be2f40b0f3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c30384d62f134e9f841056be2f40b0f3; no voters:
I20260619 16:47:55.523546 18666 leader_election.cc:288] T 00000000000000000000000000000000 P c30384d62f134e9f841056be2f40b0f3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 16:47:55.523692 18671 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c30384d62f134e9f841056be2f40b0f3 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:47:55.524329 18671 raft_consensus.cc:686] T 00000000000000000000000000000000 P c30384d62f134e9f841056be2f40b0f3 [term 1 LEADER]: Becoming Leader. State: Replica: c30384d62f134e9f841056be2f40b0f3, State: Running, Role: LEADER
I20260619 16:47:55.524806 18671 consensus_queue.cc:237] T 00000000000000000000000000000000 P c30384d62f134e9f841056be2f40b0f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c30384d62f134e9f841056be2f40b0f3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34357 } }
I20260619 16:47:55.525561 18666 sys_catalog.cc:564] T 00000000000000000000000000000000 P c30384d62f134e9f841056be2f40b0f3 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:47:55.527755 18672 sys_catalog.cc:455] T 00000000000000000000000000000000 P c30384d62f134e9f841056be2f40b0f3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c30384d62f134e9f841056be2f40b0f3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c30384d62f134e9f841056be2f40b0f3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34357 } } }
I20260619 16:47:55.527765 18673 sys_catalog.cc:455] T 00000000000000000000000000000000 P c30384d62f134e9f841056be2f40b0f3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c30384d62f134e9f841056be2f40b0f3. Latest consensus state: current_term: 1 leader_uuid: "c30384d62f134e9f841056be2f40b0f3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c30384d62f134e9f841056be2f40b0f3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34357 } } }
I20260619 16:47:55.528292 18673 sys_catalog.cc:458] T 00000000000000000000000000000000 P c30384d62f134e9f841056be2f40b0f3 [sys.catalog]: This master's current role is: LEADER
I20260619 16:47:55.528900 18672 sys_catalog.cc:458] T 00000000000000000000000000000000 P c30384d62f134e9f841056be2f40b0f3 [sys.catalog]: This master's current role is: LEADER
I20260619 16:47:55.532004 18680 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:47:55.537078 18680 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:47:55.546186 18680 catalog_manager.cc:1347] Generated new cluster ID: ffd2b6ff0ccb4a729d973d8323d35bb9
I20260619 16:47:55.546288 18680 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:47:55.560349 18680 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 16:47:55.562213 18680 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:47:55.573944 18680 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c30384d62f134e9f841056be2f40b0f3: Generated new TSK 0
I20260619 16:47:55.574934 18680 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887385903558-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887385903558-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260619 16:47:55.634610 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu
/tmp/dist-test-taskEs2EH8/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887385903558-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887385903558-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887385903558-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887385903558-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34357
--keytab_file=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887385903558-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38481
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887385903558-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887385903558-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887385903558-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:47:55.951128 18692 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:47:55.951386 18692 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:47:55.951522 18692 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:47:55.958653 18692 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:47:55.958746 18692 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:47:55.958886 18692 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260619 16:47:55.968533 18692 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38481
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887385903558-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887385903558-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887385903558-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887385903558-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34357
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887385903558-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:47:55.969801 18692 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:47:55.971056 18692 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:47:55.983703 18697 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:47:55.983665 18700 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:47:55.983665 18698 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:47:55.984467 18692 server_base.cc:1034] running on GCE node
I20260619 16:47:55.985087 18692 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:47:55.986039 18692 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:47:55.987224 18692 hybrid_clock.cc:648] HybridClock initialized: now 1781887675987168 us; error 57 us; skew 500 ppm
I20260619 16:47:55.987511 18692 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:47:55 dist-test-slave-wvz7 krb5kdc[18586](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887675, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:47:55.990550 18692 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260619 16:47:55.991933 18692 webserver.cc:460] Webserver started at http://127.0.97.1:40485/ using document root <none> and password file <none>
I20260619 16:47:55.992599 18692 fs_manager.cc:359] Metadata directory not provided
I20260619 16:47:55.992710 18692 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:47:55.992986 18692 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:47:55.995982 18692 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887385903558-388-0/minicluster-data/ts-0/data/instance:
uuid: "17071c094b8b4a3481f896f88d6a987e"
format_stamp: "Formatted at 2026-06-19 16:47:55 on dist-test-slave-wvz7"
server_key: "13a0fbab1bee5115b33dc6a28b92194f"
server_key_iv: "0802d21ce68d68b94f14f7a29ebc69b2"
server_key_version: "encryptionkey@0"
I20260619 16:47:55.996771 18692 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887385903558-388-0/minicluster-data/ts-0/wal/instance:
uuid: "17071c094b8b4a3481f896f88d6a987e"
format_stamp: "Formatted at 2026-06-19 16:47:55 on dist-test-slave-wvz7"
server_key: "13a0fbab1bee5115b33dc6a28b92194f"
server_key_iv: "0802d21ce68d68b94f14f7a29ebc69b2"
server_key_version: "encryptionkey@0"
I20260619 16:47:56.001714 18692 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.006s
I20260619 16:47:56.005285 18707 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:47:56.006625 18692 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260619 16:47:56.006780 18692 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887385903558-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887385903558-388-0/minicluster-data/ts-0/wal
uuid: "17071c094b8b4a3481f896f88d6a987e"
format_stamp: "Formatted at 2026-06-19 16:47:55 on dist-test-slave-wvz7"
server_key: "13a0fbab1bee5115b33dc6a28b92194f"
server_key_iv: "0802d21ce68d68b94f14f7a29ebc69b2"
server_key_version: "encryptionkey@0"
I20260619 16:47:56.006950 18692 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887385903558-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887385903558-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887385903558-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:47:56.028185 18692 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:47:56.032810 18692 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:47:56.033080 18692 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:47:56.034067 18692 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:47:56.035408 18692 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:47:56.035476 18692 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:47:56.035569 18692 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:47:56.035610 18692 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:47:56.063341 18692 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:42925
I20260619 16:47:56.063391 18819 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:42925 every 8 connection(s)
I20260619 16:47:56.064561 18692 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887385903558-388-0/minicluster-data/ts-0/data/info.pb
I20260619 16:47:56.069381 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu as pid 18692
I20260619 16:47:56.069545 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887385903558-388-0/minicluster-data/ts-0/wal/instance
I20260619 16:47:56.069989 388 external_mini_cluster.cc:1428] Setting key 398ad18131c47b3f9917ec88a1b83365
Jun 19 16:47:56 dist-test-slave-wvz7 krb5kdc[18586](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887675, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 16:47:56.084039 18822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:47:56.066989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:59483 (local address 127.0.97.62:34357)
0619 16:47:56.067407 (+ 418us) server_negotiation.cc:206] Beginning negotiation
0619 16:47:56.067411 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:47:56.068230 (+ 819us) server_negotiation.cc:407] Connection header received
0619 16:47:56.070108 (+ 1878us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:47:56.070113 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:47:56.070196 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:47:56.070289 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:47:56.071745 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:47:56.072427 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:47:56.074038 (+ 1611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:47:56.074452 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:47:56.074569 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:47:56.078171 (+ 3602us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:47:56.078176 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:47:56.078180 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:47:56.078211 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:47:56.081042 (+ 2831us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:47:56.081792 (+ 750us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:47:56.081798 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:47:56.081801 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:47:56.081934 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:47:56.082374 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:47:56.082378 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:47:56.082380 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:47:56.082716 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:47:56.082846 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0619 16:47:56.083473 (+ 627us) server_negotiation.cc:299] Negotiation successful
0619 16:47:56.083696 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":315,"thread_start_us":217,"threads_started":1}
I20260619 16:47:56.084815 18823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:47:56.067381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34357 (local address 127.0.97.1:59483)
0619 16:47:56.068066 (+ 685us) negotiation.cc:107] Waiting for socket to connect
0619 16:47:56.068112 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0619 16:47:56.069847 (+ 1735us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:47:56.070526 (+ 679us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:47:56.070535 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:47:56.071031 (+ 496us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:47:56.071585 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:47:56.071602 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:47:56.072562 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:47:56.072566 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:47:56.073900 (+ 1334us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:47:56.073912 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:47:56.074663 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:47:56.074671 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:47:56.074885 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:47:56.075497 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:47:56.075525 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:47:56.077954 (+ 2429us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:47:56.081179 (+ 3225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:47:56.081185 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:47:56.081198 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:47:56.081632 (+ 434us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:47:56.082063 (+ 431us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:47:56.082068 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:47:56.082072 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:47:56.082265 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:47:56.082825 (+ 560us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:47:56.082830 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:47:56.083148 (+ 318us) client_negotiation.cc:769] Sending connection context
0619 16:47:56.083362 (+ 214us) client_negotiation.cc:240] Negotiation successful
0619 16:47:56.083750 (+ 388us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":450,"thread_start_us":251,"threads_started":1}
I20260619 16:47:56.086277 18820 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34357
I20260619 16:47:56.086606 18820 heartbeater.cc:464] Registering TS with master...
I20260619 16:47:56.087384 18820 heartbeater.cc:511] Master 127.0.97.62:34357 requested a full tablet report, sending...
I20260619 16:47:56.089473 18631 ts_manager.cc:194] Registered new tserver with Master: 17071c094b8b4a3481f896f88d6a987e (127.0.97.1:42925)
I20260619 16:47:56.090942 18631 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:59483
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887385903558-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887385903558-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260619 16:47:56.120517 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu
/tmp/dist-test-taskEs2EH8/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887385903558-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887385903558-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887385903558-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887385903558-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34357
--keytab_file=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887385903558-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38481
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887385903558-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887385903558-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887385903558-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:47:56.431082 18826 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:47:56.431385 18826 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:47:56.431559 18826 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:47:56.439131 18826 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:47:56.439258 18826 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:47:56.439427 18826 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260619 16:47:56.449573 18826 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38481
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887385903558-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887385903558-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887385903558-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887385903558-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34357
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887385903558-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:47:56.450958 18826 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:47:56.452284 18826 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:47:56.465440 18834 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:47:56.465461 18832 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:47:56.465492 18831 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:47:56.466544 18826 server_base.cc:1034] running on GCE node
I20260619 16:47:56.467165 18826 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:47:56.468130 18826 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:47:56.469339 18826 hybrid_clock.cc:648] HybridClock initialized: now 1781887676469281 us; error 53 us; skew 500 ppm
I20260619 16:47:56.469702 18826 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:47:56 dist-test-slave-wvz7 krb5kdc[18586](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887676, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:47:56.472760 18826 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260619 16:47:56.474274 18826 webserver.cc:460] Webserver started at http://127.0.97.2:38061/ using document root <none> and password file <none>
I20260619 16:47:56.474942 18826 fs_manager.cc:359] Metadata directory not provided
I20260619 16:47:56.475051 18826 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:47:56.475347 18826 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:47:56.478519 18826 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887385903558-388-0/minicluster-data/ts-1/data/instance:
uuid: "2dc6ec0018f04fdfb9ae25cda2ab9650"
format_stamp: "Formatted at 2026-06-19 16:47:56 on dist-test-slave-wvz7"
server_key: "4a8253f55ce688cd9efb586668f47396"
server_key_iv: "a357a6076846c80acf623be32fa1345d"
server_key_version: "encryptionkey@0"
I20260619 16:47:56.479285 18826 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887385903558-388-0/minicluster-data/ts-1/wal/instance:
uuid: "2dc6ec0018f04fdfb9ae25cda2ab9650"
format_stamp: "Formatted at 2026-06-19 16:47:56 on dist-test-slave-wvz7"
server_key: "4a8253f55ce688cd9efb586668f47396"
server_key_iv: "a357a6076846c80acf623be32fa1345d"
server_key_version: "encryptionkey@0"
I20260619 16:47:56.484186 18826 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 16:47:56.487524 18841 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:47:56.488869 18826 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 16:47:56.489014 18826 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887385903558-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887385903558-388-0/minicluster-data/ts-1/wal
uuid: "2dc6ec0018f04fdfb9ae25cda2ab9650"
format_stamp: "Formatted at 2026-06-19 16:47:56 on dist-test-slave-wvz7"
server_key: "4a8253f55ce688cd9efb586668f47396"
server_key_iv: "a357a6076846c80acf623be32fa1345d"
server_key_version: "encryptionkey@0"
I20260619 16:47:56.489180 18826 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887385903558-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887385903558-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887385903558-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:47:56.522153 18826 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:47:56.526537 18826 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:47:56.526805 18826 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:47:56.527773 18826 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:47:56.529099 18826 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:47:56.529178 18826 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:47:56.529295 18826 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:47:56.529354 18826 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:47:56.558403 18826 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:46745
I20260619 16:47:56.558449 18953 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:46745 every 8 connection(s)
I20260619 16:47:56.559708 18826 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887385903558-388-0/minicluster-data/ts-1/data/info.pb
I20260619 16:47:56.562191 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu as pid 18826
I20260619 16:47:56.562357 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887385903558-388-0/minicluster-data/ts-1/wal/instance
I20260619 16:47:56.562777 388 external_mini_cluster.cc:1428] Setting key 60a879df76cca2e7b4d1724c42de59bc
Jun 19 16:47:56 dist-test-slave-wvz7 krb5kdc[18586](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887676, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 16:47:56.579435 18822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:47:56.562195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:36403 (local address 127.0.97.62:34357)
0619 16:47:56.562409 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0619 16:47:56.562415 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:47:56.563654 (+ 1239us) server_negotiation.cc:407] Connection header received
0619 16:47:56.565480 (+ 1826us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:47:56.565486 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:47:56.565569 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:47:56.565659 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:47:56.567210 (+ 1551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:47:56.567930 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:47:56.569489 (+ 1559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:47:56.569951 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:47:56.570062 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:47:56.574068 (+ 4006us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:47:56.574074 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:47:56.574077 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:47:56.574111 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:47:56.576566 (+ 2455us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:47:56.577237 (+ 671us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:47:56.577241 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:47:56.577243 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:47:56.577370 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:47:56.577790 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:47:56.577795 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:47:56.577798 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:47:56.578075 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:47:56.578204 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0619 16:47:56.578873 (+ 669us) server_negotiation.cc:299] Negotiation successful
0619 16:47:56.579113 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260619 16:47:56.580428 18956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:47:56.562717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34357 (local address 127.0.97.2:36403)
0619 16:47:56.563515 (+ 798us) negotiation.cc:107] Waiting for socket to connect
0619 16:47:56.563561 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0619 16:47:56.565221 (+ 1660us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:47:56.565937 (+ 716us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:47:56.565946 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:47:56.566466 (+ 520us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:47:56.567040 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:47:56.567057 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:47:56.568081 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:47:56.568085 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:47:56.569339 (+ 1254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:47:56.569351 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:47:56.570180 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:47:56.570187 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:47:56.570431 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:47:56.571099 (+ 668us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:47:56.571130 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:47:56.573829 (+ 2699us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:47:56.576697 (+ 2868us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:47:56.576707 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:47:56.576725 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:47:56.577125 (+ 400us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:47:56.577475 (+ 350us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:47:56.577480 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:47:56.577484 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:47:56.577669 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:47:56.578191 (+ 522us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:47:56.578198 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:47:56.578534 (+ 336us) client_negotiation.cc:769] Sending connection context
0619 16:47:56.578776 (+ 242us) client_negotiation.cc:240] Negotiation successful
0619 16:47:56.579304 (+ 528us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":490,"thread_start_us":320,"threads_started":1}
I20260619 16:47:56.581845 18954 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34357
I20260619 16:47:56.582134 18954 heartbeater.cc:464] Registering TS with master...
I20260619 16:47:56.582832 18954 heartbeater.cc:511] Master 127.0.97.62:34357 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260619 16:47:56.584326 18631 ts_manager.cc:194] Registered new tserver with Master: 2dc6ec0018f04fdfb9ae25cda2ab9650 (127.0.97.2:46745)
I20260619 16:47:56.585186 18631 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:36403
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887385903558-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887385903558-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260619 16:47:56.612428 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu
/tmp/dist-test-taskEs2EH8/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887385903558-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887385903558-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887385903558-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887385903558-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34357
--keytab_file=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887385903558-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38481
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887385903558-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887385903558-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887385903558-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:47:56.921757 18959 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:47:56.922026 18959 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:47:56.922115 18959 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:47:56.929394 18959 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:47:56.929502 18959 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:47:56.929620 18959 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260619 16:47:56.939417 18959 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38481
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887385903558-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887385903558-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887385903558-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887385903558-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34357
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887385903558-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:47:56.940589 18959 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:47:56.941836 18959 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:47:56.954016 18965 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:47:56.954149 18967 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:47:56.954149 18964 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:47:56.955030 18959 server_base.cc:1034] running on GCE node
I20260619 16:47:56.955590 18959 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:47:56.956439 18959 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:47:56.957636 18959 hybrid_clock.cc:648] HybridClock initialized: now 1781887676957594 us; error 31 us; skew 500 ppm
I20260619 16:47:56.957962 18959 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:47:56 dist-test-slave-wvz7 krb5kdc[18586](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887676, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:47:56.960968 18959 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260619 16:47:56.962354 18959 webserver.cc:460] Webserver started at http://127.0.97.3:40015/ using document root <none> and password file <none>
I20260619 16:47:56.963017 18959 fs_manager.cc:359] Metadata directory not provided
I20260619 16:47:56.963145 18959 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:47:56.963423 18959 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:47:56.966262 18959 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887385903558-388-0/minicluster-data/ts-2/data/instance:
uuid: "ba3be5462aef45668ff34dd3e7fce2c9"
format_stamp: "Formatted at 2026-06-19 16:47:56 on dist-test-slave-wvz7"
server_key: "1ea4f02ea1c2d3050d524d36b86f60c0"
server_key_iv: "31f43080c3656060b293bbb179ffc391"
server_key_version: "encryptionkey@0"
I20260619 16:47:56.967028 18959 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887385903558-388-0/minicluster-data/ts-2/wal/instance:
uuid: "ba3be5462aef45668ff34dd3e7fce2c9"
format_stamp: "Formatted at 2026-06-19 16:47:56 on dist-test-slave-wvz7"
server_key: "1ea4f02ea1c2d3050d524d36b86f60c0"
server_key_iv: "31f43080c3656060b293bbb179ffc391"
server_key_version: "encryptionkey@0"
I20260619 16:47:56.972011 18959 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 16:47:56.975310 18974 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:47:56.976553 18959 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 16:47:56.976698 18959 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887385903558-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887385903558-388-0/minicluster-data/ts-2/wal
uuid: "ba3be5462aef45668ff34dd3e7fce2c9"
format_stamp: "Formatted at 2026-06-19 16:47:56 on dist-test-slave-wvz7"
server_key: "1ea4f02ea1c2d3050d524d36b86f60c0"
server_key_iv: "31f43080c3656060b293bbb179ffc391"
server_key_version: "encryptionkey@0"
I20260619 16:47:56.976886 18959 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887385903558-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887385903558-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887385903558-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:47:56.990718 18959 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:47:56.994844 18959 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:47:56.995113 18959 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:47:56.996074 18959 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:47:56.997352 18959 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:47:56.997427 18959 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:47:56.997538 18959 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:47:56.997593 18959 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:47:57.026871 18959 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:41543
I20260619 16:47:57.026908 19086 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:41543 every 8 connection(s)
I20260619 16:47:57.028100 18959 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887385903558-388-0/minicluster-data/ts-2/data/info.pb
I20260619 16:47:57.034602 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu as pid 18959
I20260619 16:47:57.034809 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887385903558-388-0/minicluster-data/ts-2/wal/instance
I20260619 16:47:57.035255 388 external_mini_cluster.cc:1428] Setting key 348eda048be8f92f2778671c92454aea
Jun 19 16:47:57 dist-test-slave-wvz7 krb5kdc[18586](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887676, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 16:47:57.048419 18822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:47:57.030793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:46101 (local address 127.0.97.62:34357)
0619 16:47:57.031012 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0619 16:47:57.031016 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:47:57.032153 (+ 1137us) server_negotiation.cc:407] Connection header received
0619 16:47:57.034045 (+ 1892us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:47:57.034051 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:47:57.034151 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:47:57.034255 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:47:57.035951 (+ 1696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:47:57.037030 (+ 1079us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:47:57.038960 (+ 1930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:47:57.039385 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:47:57.039486 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:47:57.042939 (+ 3453us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:47:57.042944 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:47:57.042948 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:47:57.042983 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:47:57.045428 (+ 2445us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:47:57.046210 (+ 782us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:47:57.046214 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:47:57.046217 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:47:57.046339 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:47:57.046816 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:47:57.046821 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:47:57.046823 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:47:57.047127 (+ 304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:47:57.047251 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0619 16:47:57.047941 (+ 690us) server_negotiation.cc:299] Negotiation successful
0619 16:47:57.048120 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97}
I20260619 16:47:57.049368 19089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:47:57.031226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34357 (local address 127.0.97.3:46101)
0619 16:47:57.031957 (+ 731us) negotiation.cc:107] Waiting for socket to connect
0619 16:47:57.032045 (+ 88us) client_negotiation.cc:174] Beginning negotiation
0619 16:47:57.033780 (+ 1735us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:47:57.034499 (+ 719us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:47:57.034508 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:47:57.035087 (+ 579us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:47:57.035742 (+ 655us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:47:57.035768 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:47:57.037195 (+ 1427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:47:57.037201 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:47:57.038821 (+ 1620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:47:57.038833 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:47:57.039594 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:47:57.039599 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:47:57.039792 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:47:57.040381 (+ 589us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:47:57.040403 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:47:57.042717 (+ 2314us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:47:57.045580 (+ 2863us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:47:57.045587 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:47:57.045600 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:47:57.046112 (+ 512us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:47:57.046463 (+ 351us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:47:57.046466 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:47:57.046488 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:47:57.046702 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:47:57.047246 (+ 544us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:47:57.047252 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:47:57.047567 (+ 315us) client_negotiation.cc:769] Sending connection context
0619 16:47:57.047786 (+ 219us) client_negotiation.cc:240] Negotiation successful
0619 16:47:57.048190 (+ 404us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":503,"thread_start_us":310,"threads_started":1}
I20260619 16:47:57.050913 19087 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34357
I20260619 16:47:57.051208 19087 heartbeater.cc:464] Registering TS with master...
I20260619 16:47:57.051895 19087 heartbeater.cc:511] Master 127.0.97.62:34357 requested a full tablet report, sending...
I20260619 16:47:57.053231 18631 ts_manager.cc:194] Registered new tserver with Master: ba3be5462aef45668ff34dd3e7fce2c9 (127.0.97.3:41543)
I20260619 16:47:57.054064 18631 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:46101
I20260619 16:47:57.062690 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 16:47:57.079504 18822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:47:57.067613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49232 (local address 127.0.97.62:34357)
0619 16:47:57.067786 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0619 16:47:57.067792 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:47:57.068193 (+ 401us) server_negotiation.cc:407] Connection header received
0619 16:47:57.068308 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:47:57.068313 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:47:57.068400 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:47:57.068503 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:47:57.069823 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:47:57.070543 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:47:57.071778 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:47:57.072179 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:47:57.072271 (+ 92us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:47:57.074687 (+ 2416us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:47:57.074693 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:47:57.074696 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:47:57.074729 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:47:57.077128 (+ 2399us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:47:57.077841 (+ 713us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:47:57.077846 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:47:57.077848 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:47:57.077970 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:47:57.078296 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:47:57.078299 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:47:57.078301 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:47:57.078625 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:47:57.078819 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0619 16:47:57.079103 (+ 284us) server_negotiation.cc:299] Negotiation successful
0619 16:47:57.079253 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260619 16:47:57.083169 18631 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49232:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 16:47:57.089990 18631 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 16:47:57.094266 18820 heartbeater.cc:503] Master 127.0.97.62:34357 was elected leader, sending a full tablet report...
I20260619 16:47:57.111799 19099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:47:57.101896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:41543 (local address 127.0.0.1:51578)
0619 16:47:57.102717 (+ 821us) negotiation.cc:107] Waiting for socket to connect
0619 16:47:57.102752 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0619 16:47:57.102937 (+ 185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:47:57.103890 (+ 953us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:47:57.103898 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:47:57.103927 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:47:57.104332 (+ 405us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:47:57.104346 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:47:57.106290 (+ 1944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:47:57.106295 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:47:57.108685 (+ 2390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:47:57.108700 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:47:57.110796 (+ 2096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:47:57.110802 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:47:57.111005 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:47:57.111061 (+ 56us) client_negotiation.cc:769] Sending connection context
0619 16:47:57.111182 (+ 121us) client_negotiation.cc:240] Negotiation successful
0619 16:47:57.111367 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":584,"mutex_wait_us":23,"thread_start_us":435,"threads_started":1}
I20260619 16:47:57.112710 19100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:47:57.102159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51578 (local address 127.0.97.3:41543)
0619 16:47:57.102813 (+ 654us) server_negotiation.cc:206] Beginning negotiation
0619 16:47:57.102819 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:47:57.102846 (+ 27us) server_negotiation.cc:407] Connection header received
0619 16:47:57.103140 (+ 294us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:47:57.103147 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:47:57.103401 (+ 254us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:47:57.103602 (+ 201us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:47:57.104534 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:47:57.106118 (+ 1584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:47:57.108875 (+ 2757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:47:57.110293 (+ 1418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:47:57.110410 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:47:57.110637 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0619 16:47:57.112228 (+ 1591us) server_negotiation.cc:299] Negotiation successful
0619 16:47:57.112392 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":521,"thread_start_us":374,"threads_started":1}
I20260619 16:47:57.112788 19102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:47:57.103500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59914 (local address 127.0.97.1:42925)
0619 16:47:57.104233 (+ 733us) server_negotiation.cc:206] Beginning negotiation
0619 16:47:57.104241 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 16:47:57.104267 (+ 26us) server_negotiation.cc:407] Connection header received
0619 16:47:57.104355 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:47:57.104362 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:47:57.104612 (+ 250us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:47:57.105416 (+ 804us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:47:57.105649 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:47:57.107123 (+ 1474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:47:57.109491 (+ 2368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:47:57.110967 (+ 1476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:47:57.112081 (+ 1114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:47:57.112303 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0619 16:47:57.112366 (+ 63us) server_negotiation.cc:299] Negotiation successful
0619 16:47:57.112513 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":579,"thread_start_us":422,"threads_started":1}
I20260619 16:47:57.111949 19101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:47:57.103130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42925 (local address 127.0.0.1:59914)
0619 16:47:57.103620 (+ 490us) negotiation.cc:107] Waiting for socket to connect
0619 16:47:57.103636 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 16:47:57.103741 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:47:57.104817 (+ 1076us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:47:57.104821 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:47:57.104845 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:47:57.105235 (+ 390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:47:57.105248 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:47:57.107271 (+ 2023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:47:57.107276 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:47:57.109286 (+ 2010us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:47:57.109300 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:47:57.111180 (+ 1880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:47:57.111187 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:47:57.111350 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:47:57.111378 (+ 28us) client_negotiation.cc:769] Sending connection context
0619 16:47:57.111433 (+ 55us) client_negotiation.cc:240] Negotiation successful
0619 16:47:57.111564 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":353,"mutex_wait_us":1,"thread_start_us":239,"threads_started":1}
I20260619 16:47:57.117959 18755 tablet_service.cc:1467] Processing CreateTablet for tablet 93a1361aecb142b88cf7e500dcdd780d (DEFAULT_TABLE table=test-table [id=ce4726a641064bd5ae636b051b65d11a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:47:57.118526 19022 tablet_service.cc:1467] Processing CreateTablet for tablet 93a1361aecb142b88cf7e500dcdd780d (DEFAULT_TABLE table=test-table [id=ce4726a641064bd5ae636b051b65d11a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:47:57.119486 19099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:47:57.104278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:46745 (local address 127.0.0.1:49662)
0619 16:47:57.113257 (+ 8979us) negotiation.cc:107] Waiting for socket to connect
0619 16:47:57.113274 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 16:47:57.113371 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:47:57.114128 (+ 757us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:47:57.114132 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:47:57.114157 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:47:57.114320 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:47:57.114329 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:47:57.115802 (+ 1473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:47:57.115807 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:47:57.116971 (+ 1164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:47:57.116984 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:47:57.118791 (+ 1807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:47:57.118797 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:47:57.118934 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:47:57.118956 (+ 22us) client_negotiation.cc:769] Sending connection context
0619 16:47:57.119036 (+ 80us) client_negotiation.cc:240] Negotiation successful
0619 16:47:57.119177 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8878,"thread_start_us":9419,"threads_started":1}
I20260619 16:47:57.119522 19103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:47:57.103926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49662 (local address 127.0.97.2:46745)
0619 16:47:57.113592 (+ 9666us) server_negotiation.cc:206] Beginning negotiation
0619 16:47:57.113600 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 16:47:57.113628 (+ 28us) server_negotiation.cc:407] Connection header received
0619 16:47:57.113712 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:47:57.113717 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:47:57.113961 (+ 244us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:47:57.114127 (+ 166us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:47:57.114445 (+ 318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:47:57.115661 (+ 1216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:47:57.117123 (+ 1462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:47:57.118384 (+ 1261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:47:57.118517 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:47:57.118699 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0619 16:47:57.119097 (+ 398us) server_negotiation.cc:299] Negotiation successful
0619 16:47:57.119256 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9491,"thread_start_us":9391,"threads_started":1}
I20260619 16:47:57.119593 18755 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 93a1361aecb142b88cf7e500dcdd780d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:47:57.120149 19022 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 93a1361aecb142b88cf7e500dcdd780d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:47:57.122670 18889 tablet_service.cc:1467] Processing CreateTablet for tablet 93a1361aecb142b88cf7e500dcdd780d (DEFAULT_TABLE table=test-table [id=ce4726a641064bd5ae636b051b65d11a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:47:57.123703 18889 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 93a1361aecb142b88cf7e500dcdd780d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:47:57.133985 19105 tablet_bootstrap.cc:492] T 93a1361aecb142b88cf7e500dcdd780d P 17071c094b8b4a3481f896f88d6a987e: Bootstrap starting.
I20260619 16:47:57.138118 19105 tablet_bootstrap.cc:654] T 93a1361aecb142b88cf7e500dcdd780d P 17071c094b8b4a3481f896f88d6a987e: Neither blocks nor log segments found. Creating new log.
I20260619 16:47:57.139592 19105 log.cc:824] T 93a1361aecb142b88cf7e500dcdd780d P 17071c094b8b4a3481f896f88d6a987e: Log is configured to *not* fsync() on all Append() calls
I20260619 16:47:57.139895 19106 tablet_bootstrap.cc:492] T 93a1361aecb142b88cf7e500dcdd780d P ba3be5462aef45668ff34dd3e7fce2c9: Bootstrap starting.
I20260619 16:47:57.143119 19106 tablet_bootstrap.cc:654] T 93a1361aecb142b88cf7e500dcdd780d P ba3be5462aef45668ff34dd3e7fce2c9: Neither blocks nor log segments found. Creating new log.
I20260619 16:47:57.143543 19105 tablet_bootstrap.cc:492] T 93a1361aecb142b88cf7e500dcdd780d P 17071c094b8b4a3481f896f88d6a987e: No bootstrap required, opened a new log
I20260619 16:47:57.143889 19105 ts_tablet_manager.cc:1397] T 93a1361aecb142b88cf7e500dcdd780d P 17071c094b8b4a3481f896f88d6a987e: Time spent bootstrapping tablet: real 0.010s user 0.009s sys 0.000s
I20260619 16:47:57.143997 19107 tablet_bootstrap.cc:492] T 93a1361aecb142b88cf7e500dcdd780d P 2dc6ec0018f04fdfb9ae25cda2ab9650: Bootstrap starting.
I20260619 16:47:57.144313 19106 log.cc:824] T 93a1361aecb142b88cf7e500dcdd780d P ba3be5462aef45668ff34dd3e7fce2c9: Log is configured to *not* fsync() on all Append() calls
I20260619 16:47:57.148036 19107 tablet_bootstrap.cc:654] T 93a1361aecb142b88cf7e500dcdd780d P 2dc6ec0018f04fdfb9ae25cda2ab9650: Neither blocks nor log segments found. Creating new log.
I20260619 16:47:57.149350 19107 log.cc:824] T 93a1361aecb142b88cf7e500dcdd780d P 2dc6ec0018f04fdfb9ae25cda2ab9650: Log is configured to *not* fsync() on all Append() calls
I20260619 16:47:57.152056 19107 tablet_bootstrap.cc:492] T 93a1361aecb142b88cf7e500dcdd780d P 2dc6ec0018f04fdfb9ae25cda2ab9650: No bootstrap required, opened a new log
I20260619 16:47:57.152288 19107 ts_tablet_manager.cc:1397] T 93a1361aecb142b88cf7e500dcdd780d P 2dc6ec0018f04fdfb9ae25cda2ab9650: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.003s
I20260619 16:47:57.152432 19106 tablet_bootstrap.cc:492] T 93a1361aecb142b88cf7e500dcdd780d P ba3be5462aef45668ff34dd3e7fce2c9: No bootstrap required, opened a new log
I20260619 16:47:57.152702 19106 ts_tablet_manager.cc:1397] T 93a1361aecb142b88cf7e500dcdd780d P ba3be5462aef45668ff34dd3e7fce2c9: Time spent bootstrapping tablet: real 0.013s user 0.007s sys 0.004s
I20260619 16:47:57.152555 19105 raft_consensus.cc:348] T 93a1361aecb142b88cf7e500dcdd780d P 17071c094b8b4a3481f896f88d6a987e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba3be5462aef45668ff34dd3e7fce2c9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41543 } } peers { permanent_uuid: "17071c094b8b4a3481f896f88d6a987e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42925 } } peers { permanent_uuid: "2dc6ec0018f04fdfb9ae25cda2ab9650" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46745 } }
I20260619 16:47:57.152974 19105 raft_consensus.cc:374] T 93a1361aecb142b88cf7e500dcdd780d P 17071c094b8b4a3481f896f88d6a987e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:47:57.153074 19105 raft_consensus.cc:729] T 93a1361aecb142b88cf7e500dcdd780d P 17071c094b8b4a3481f896f88d6a987e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17071c094b8b4a3481f896f88d6a987e, State: Initialized, Role: FOLLOWER
I20260619 16:47:57.153680 19105 consensus_queue.cc:260] T 93a1361aecb142b88cf7e500dcdd780d P 17071c094b8b4a3481f896f88d6a987e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba3be5462aef45668ff34dd3e7fce2c9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41543 } } peers { permanent_uuid: "17071c094b8b4a3481f896f88d6a987e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42925 } } peers { permanent_uuid: "2dc6ec0018f04fdfb9ae25cda2ab9650" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46745 } }
I20260619 16:47:57.159245 19107 raft_consensus.cc:348] T 93a1361aecb142b88cf7e500dcdd780d P 2dc6ec0018f04fdfb9ae25cda2ab9650 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba3be5462aef45668ff34dd3e7fce2c9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41543 } } peers { permanent_uuid: "17071c094b8b4a3481f896f88d6a987e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42925 } } peers { permanent_uuid: "2dc6ec0018f04fdfb9ae25cda2ab9650" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46745 } }
I20260619 16:47:57.159590 19107 raft_consensus.cc:374] T 93a1361aecb142b88cf7e500dcdd780d P 2dc6ec0018f04fdfb9ae25cda2ab9650 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:47:57.159725 19107 raft_consensus.cc:729] T 93a1361aecb142b88cf7e500dcdd780d P 2dc6ec0018f04fdfb9ae25cda2ab9650 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2dc6ec0018f04fdfb9ae25cda2ab9650, State: Initialized, Role: FOLLOWER
I20260619 16:47:57.160401 19107 consensus_queue.cc:260] T 93a1361aecb142b88cf7e500dcdd780d P 2dc6ec0018f04fdfb9ae25cda2ab9650 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba3be5462aef45668ff34dd3e7fce2c9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41543 } } peers { permanent_uuid: "17071c094b8b4a3481f896f88d6a987e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42925 } } peers { permanent_uuid: "2dc6ec0018f04fdfb9ae25cda2ab9650" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46745 } }
I20260619 16:47:57.161787 19105 ts_tablet_manager.cc:1428] T 93a1361aecb142b88cf7e500dcdd780d P 17071c094b8b4a3481f896f88d6a987e: Time spent starting tablet: real 0.018s user 0.013s sys 0.004s
I20260619 16:47:57.162704 18954 heartbeater.cc:503] Master 127.0.97.62:34357 was elected leader, sending a full tablet report...
I20260619 16:47:57.163216 19107 ts_tablet_manager.cc:1428] T 93a1361aecb142b88cf7e500dcdd780d P 2dc6ec0018f04fdfb9ae25cda2ab9650: Time spent starting tablet: real 0.011s user 0.011s sys 0.001s
I20260619 16:47:57.164075 19106 raft_consensus.cc:348] T 93a1361aecb142b88cf7e500dcdd780d P ba3be5462aef45668ff34dd3e7fce2c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba3be5462aef45668ff34dd3e7fce2c9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41543 } } peers { permanent_uuid: "17071c094b8b4a3481f896f88d6a987e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42925 } } peers { permanent_uuid: "2dc6ec0018f04fdfb9ae25cda2ab9650" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46745 } }
I20260619 16:47:57.164548 19106 raft_consensus.cc:374] T 93a1361aecb142b88cf7e500dcdd780d P ba3be5462aef45668ff34dd3e7fce2c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:47:57.164642 19106 raft_consensus.cc:729] T 93a1361aecb142b88cf7e500dcdd780d P ba3be5462aef45668ff34dd3e7fce2c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba3be5462aef45668ff34dd3e7fce2c9, State: Initialized, Role: FOLLOWER
I20260619 16:47:57.165306 19106 consensus_queue.cc:260] T 93a1361aecb142b88cf7e500dcdd780d P ba3be5462aef45668ff34dd3e7fce2c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba3be5462aef45668ff34dd3e7fce2c9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41543 } } peers { permanent_uuid: "17071c094b8b4a3481f896f88d6a987e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42925 } } peers { permanent_uuid: "2dc6ec0018f04fdfb9ae25cda2ab9650" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46745 } }
I20260619 16:47:57.166343 19087 heartbeater.cc:503] Master 127.0.97.62:34357 was elected leader, sending a full tablet report...
I20260619 16:47:57.167878 19106 ts_tablet_manager.cc:1428] T 93a1361aecb142b88cf7e500dcdd780d P ba3be5462aef45668ff34dd3e7fce2c9: Time spent starting tablet: real 0.015s user 0.009s sys 0.006s
I20260619 16:47:57.177582 19112 raft_consensus.cc:482] T 93a1361aecb142b88cf7e500dcdd780d P 2dc6ec0018f04fdfb9ae25cda2ab9650 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 16:47:57.177937 19112 raft_consensus.cc:504] T 93a1361aecb142b88cf7e500dcdd780d P 2dc6ec0018f04fdfb9ae25cda2ab9650 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba3be5462aef45668ff34dd3e7fce2c9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41543 } } peers { permanent_uuid: "17071c094b8b4a3481f896f88d6a987e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42925 } } peers { permanent_uuid: "2dc6ec0018f04fdfb9ae25cda2ab9650" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46745 } }
I20260619 16:47:57.179544 19112 leader_election.cc:288] T 93a1361aecb142b88cf7e500dcdd780d P 2dc6ec0018f04fdfb9ae25cda2ab9650 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ba3be5462aef45668ff34dd3e7fce2c9 (127.0.97.3:41543), 17071c094b8b4a3481f896f88d6a987e (127.0.97.1:42925)
I20260619 16:47:57.188192 19102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:47:57.180823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:45495 (local address 127.0.97.1:42925)
0619 16:47:57.181070 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0619 16:47:57.181075 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:47:57.181206 (+ 131us) server_negotiation.cc:407] Connection header received
0619 16:47:57.181364 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:47:57.181367 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:47:57.181423 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:47:57.181520 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:47:57.182086 (+ 566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:47:57.183117 (+ 1031us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:47:57.185855 (+ 2738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:47:57.187083 (+ 1228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:47:57.187213 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:47:57.187317 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0619 16:47:57.187815 (+ 498us) server_negotiation.cc:299] Negotiation successful
0619 16:47:57.187946 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123}
I20260619 16:47:57.188191 19100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:47:57.179975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:48587 (local address 127.0.97.3:41543)
0619 16:47:57.180219 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0619 16:47:57.180226 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:47:57.180613 (+ 387us) server_negotiation.cc:407] Connection header received
0619 16:47:57.180914 (+ 301us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:47:57.180920 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:47:57.180997 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:47:57.181061 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:47:57.181889 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:47:57.183019 (+ 1130us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:47:57.185855 (+ 2836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:47:57.187083 (+ 1228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:47:57.187213 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:47:57.187318 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0619 16:47:57.187815 (+ 497us) server_negotiation.cc:299] Negotiation successful
0619 16:47:57.187946 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260619 16:47:57.188572 19114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:47:57.179863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:41543 (local address 127.0.97.2:48587)
0619 16:47:57.180508 (+ 645us) negotiation.cc:107] Waiting for socket to connect
0619 16:47:57.180532 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0619 16:47:57.180734 (+ 202us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:47:57.181273 (+ 539us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:47:57.181277 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:47:57.181298 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:47:57.181751 (+ 453us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:47:57.181762 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:47:57.183177 (+ 1415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:47:57.183184 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:47:57.185692 (+ 2508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:47:57.185702 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:47:57.187366 (+ 1664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:47:57.187373 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:47:57.187617 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:47:57.187671 (+ 54us) client_negotiation.cc:769] Sending connection context
0619 16:47:57.187766 (+ 95us) client_negotiation.cc:240] Negotiation successful
0619 16:47:57.187920 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":507,"mutex_wait_us":145,"thread_start_us":348,"threads_started":1}
I20260619 16:47:57.188637 19115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:47:57.180552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42925 (local address 127.0.97.2:45495)
0619 16:47:57.181135 (+ 583us) negotiation.cc:107] Waiting for socket to connect
0619 16:47:57.181151 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 16:47:57.181262 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:47:57.181538 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:47:57.181542 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:47:57.181562 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:47:57.181934 (+ 372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:47:57.181943 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:47:57.183242 (+ 1299us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:47:57.183246 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:47:57.185673 (+ 2427us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:47:57.185692 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:47:57.187370 (+ 1678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:47:57.187375 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:47:57.187615 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:47:57.187677 (+ 62us) client_negotiation.cc:769] Sending connection context
0619 16:47:57.187766 (+ 89us) client_negotiation.cc:240] Negotiation successful
0619 16:47:57.187923 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":472,"mutex_wait_us":149,"thread_start_us":253,"threads_started":1}
I20260619 16:47:57.189671 19042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93a1361aecb142b88cf7e500dcdd780d" candidate_uuid: "2dc6ec0018f04fdfb9ae25cda2ab9650" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba3be5462aef45668ff34dd3e7fce2c9" is_pre_election: true
I20260619 16:47:57.189671 18775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93a1361aecb142b88cf7e500dcdd780d" candidate_uuid: "2dc6ec0018f04fdfb9ae25cda2ab9650" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17071c094b8b4a3481f896f88d6a987e" is_pre_election: true
I20260619 16:47:57.190261 19042 raft_consensus.cc:2454] T 93a1361aecb142b88cf7e500dcdd780d P ba3be5462aef45668ff34dd3e7fce2c9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2dc6ec0018f04fdfb9ae25cda2ab9650 in term 0.
I20260619 16:47:57.190261 18775 raft_consensus.cc:2454] T 93a1361aecb142b88cf7e500dcdd780d P 17071c094b8b4a3481f896f88d6a987e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2dc6ec0018f04fdfb9ae25cda2ab9650 in term 0.
I20260619 16:47:57.190903 18845 leader_election.cc:302] T 93a1361aecb142b88cf7e500dcdd780d P 2dc6ec0018f04fdfb9ae25cda2ab9650 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2dc6ec0018f04fdfb9ae25cda2ab9650, ba3be5462aef45668ff34dd3e7fce2c9; no voters:
I20260619 16:47:57.191275 19112 raft_consensus.cc:2781] T 93a1361aecb142b88cf7e500dcdd780d P 2dc6ec0018f04fdfb9ae25cda2ab9650 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 16:47:57.191421 19112 raft_consensus.cc:482] T 93a1361aecb142b88cf7e500dcdd780d P 2dc6ec0018f04fdfb9ae25cda2ab9650 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 16:47:57.191524 19112 raft_consensus.cc:3037] T 93a1361aecb142b88cf7e500dcdd780d P 2dc6ec0018f04fdfb9ae25cda2ab9650 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:47:57.194137 19112 raft_consensus.cc:504] T 93a1361aecb142b88cf7e500dcdd780d P 2dc6ec0018f04fdfb9ae25cda2ab9650 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba3be5462aef45668ff34dd3e7fce2c9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41543 } } peers { permanent_uuid: "17071c094b8b4a3481f896f88d6a987e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42925 } } peers { permanent_uuid: "2dc6ec0018f04fdfb9ae25cda2ab9650" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46745 } }
I20260619 16:47:57.194715 19112 leader_election.cc:288] T 93a1361aecb142b88cf7e500dcdd780d P 2dc6ec0018f04fdfb9ae25cda2ab9650 [CANDIDATE]: Term 1 election: Requested vote from peers ba3be5462aef45668ff34dd3e7fce2c9 (127.0.97.3:41543), 17071c094b8b4a3481f896f88d6a987e (127.0.97.1:42925)
I20260619 16:47:57.195226 19042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93a1361aecb142b88cf7e500dcdd780d" candidate_uuid: "2dc6ec0018f04fdfb9ae25cda2ab9650" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba3be5462aef45668ff34dd3e7fce2c9"
I20260619 16:47:57.195233 18775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93a1361aecb142b88cf7e500dcdd780d" candidate_uuid: "2dc6ec0018f04fdfb9ae25cda2ab9650" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17071c094b8b4a3481f896f88d6a987e"
I20260619 16:47:57.195447 18775 raft_consensus.cc:3037] T 93a1361aecb142b88cf7e500dcdd780d P 17071c094b8b4a3481f896f88d6a987e [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:47:57.195483 19042 raft_consensus.cc:3037] T 93a1361aecb142b88cf7e500dcdd780d P ba3be5462aef45668ff34dd3e7fce2c9 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:47:57.198586 18775 raft_consensus.cc:2454] T 93a1361aecb142b88cf7e500dcdd780d P 17071c094b8b4a3481f896f88d6a987e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2dc6ec0018f04fdfb9ae25cda2ab9650 in term 1.
I20260619 16:47:57.198583 19042 raft_consensus.cc:2454] T 93a1361aecb142b88cf7e500dcdd780d P ba3be5462aef45668ff34dd3e7fce2c9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2dc6ec0018f04fdfb9ae25cda2ab9650 in term 1.
I20260619 16:47:57.198977 18845 leader_election.cc:302] T 93a1361aecb142b88cf7e500dcdd780d P 2dc6ec0018f04fdfb9ae25cda2ab9650 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17071c094b8b4a3481f896f88d6a987e, 2dc6ec0018f04fdfb9ae25cda2ab9650; no voters:
I20260619 16:47:57.199277 19112 raft_consensus.cc:2781] T 93a1361aecb142b88cf7e500dcdd780d P 2dc6ec0018f04fdfb9ae25cda2ab9650 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:47:57.199715 19112 raft_consensus.cc:686] T 93a1361aecb142b88cf7e500dcdd780d P 2dc6ec0018f04fdfb9ae25cda2ab9650 [term 1 LEADER]: Becoming Leader. State: Replica: 2dc6ec0018f04fdfb9ae25cda2ab9650, State: Running, Role: LEADER
I20260619 16:47:57.200256 19112 consensus_queue.cc:237] T 93a1361aecb142b88cf7e500dcdd780d P 2dc6ec0018f04fdfb9ae25cda2ab9650 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba3be5462aef45668ff34dd3e7fce2c9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41543 } } peers { permanent_uuid: "17071c094b8b4a3481f896f88d6a987e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42925 } } peers { permanent_uuid: "2dc6ec0018f04fdfb9ae25cda2ab9650" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46745 } }
I20260619 16:47:57.204710 18630 catalog_manager.cc:5495] T 93a1361aecb142b88cf7e500dcdd780d P 2dc6ec0018f04fdfb9ae25cda2ab9650 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2dc6ec0018f04fdfb9ae25cda2ab9650 (127.0.97.2). New cstate: current_term: 1 leader_uuid: "2dc6ec0018f04fdfb9ae25cda2ab9650" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba3be5462aef45668ff34dd3e7fce2c9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41543 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "17071c094b8b4a3481f896f88d6a987e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42925 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2dc6ec0018f04fdfb9ae25cda2ab9650" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46745 } health_report { overall_health: HEALTHY } } }
I20260619 16:47:57.225595 19103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:47:57.220294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49674 (local address 127.0.97.2:46745)
0619 16:47:57.220491 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0619 16:47:57.220496 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:47:57.220527 (+ 31us) server_negotiation.cc:407] Connection header received
0619 16:47:57.220648 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:47:57.220652 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:47:57.220728 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:47:57.220858 (+ 130us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 16:47:57.221244 (+ 386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:47:57.222005 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:47:57.223347 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:47:57.223748 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:47:57.223855 (+ 107us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:47:57.224469 (+ 614us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 16:47:57.225010 (+ 541us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 16:47:57.225174 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0619 16:47:57.225319 (+ 145us) server_negotiation.cc:299] Negotiation successful
0619 16:47:57.225364 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260619 16:47:57.233958 18775 raft_consensus.cc:1261] T 93a1361aecb142b88cf7e500dcdd780d P 17071c094b8b4a3481f896f88d6a987e [term 1 FOLLOWER]: Refusing update from remote peer 2dc6ec0018f04fdfb9ae25cda2ab9650: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:47:57.234061 19042 raft_consensus.cc:1261] T 93a1361aecb142b88cf7e500dcdd780d P ba3be5462aef45668ff34dd3e7fce2c9 [term 1 FOLLOWER]: Refusing update from remote peer 2dc6ec0018f04fdfb9ae25cda2ab9650: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:47:57.234861 19116 consensus_queue.cc:1035] T 93a1361aecb142b88cf7e500dcdd780d P 2dc6ec0018f04fdfb9ae25cda2ab9650 [LEADER]: Connected to new peer: Peer: permanent_uuid: "17071c094b8b4a3481f896f88d6a987e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42925 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:47:57.235261 19112 consensus_queue.cc:1035] T 93a1361aecb142b88cf7e500dcdd780d P 2dc6ec0018f04fdfb9ae25cda2ab9650 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba3be5462aef45668ff34dd3e7fce2c9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41543 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:47:57.245775 19119 mvcc.cc:205] Tried to move back new op lower bound from 7298611925939224576 to 7298611925816315904. Current Snapshot: MvccSnapshot[applied={T|T < 7298611925939224576}]
I20260619 16:47:57.259464 18630 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49232:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:47:57.259765 18630 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49232:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:47:57.263500 18630 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c30384d62f134e9f841056be2f40b0f3: Sending DeleteTablet for 3 replicas of tablet 93a1361aecb142b88cf7e500dcdd780d
I20260619 16:47:57.264634 19022 tablet_service.cc:1514] Processing DeleteTablet for tablet 93a1361aecb142b88cf7e500dcdd780d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:47:57 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:51578
I20260619 16:47:57.264914 18755 tablet_service.cc:1514] Processing DeleteTablet for tablet 93a1361aecb142b88cf7e500dcdd780d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:47:57 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:59914
I20260619 16:47:57.265097 18889 tablet_service.cc:1514] Processing DeleteTablet for tablet 93a1361aecb142b88cf7e500dcdd780d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:47:57 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:49662
I20260619 16:47:57.268546 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu pid 18692
I20260619 16:47:57.275892 19130 tablet_replica.cc:330] T 93a1361aecb142b88cf7e500dcdd780d P 17071c094b8b4a3481f896f88d6a987e: stopping tablet replica
I20260619 16:47:57.277009 19130 raft_consensus.cc:2229] T 93a1361aecb142b88cf7e500dcdd780d P 17071c094b8b4a3481f896f88d6a987e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 16:47:57.278255 19130 raft_consensus.cc:2258] T 93a1361aecb142b88cf7e500dcdd780d P 17071c094b8b4a3481f896f88d6a987e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:47:57.281849 19130 ts_tablet_manager.cc:1905] T 93a1361aecb142b88cf7e500dcdd780d P 17071c094b8b4a3481f896f88d6a987e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:47:57.283315 19131 tablet_replica.cc:330] T 93a1361aecb142b88cf7e500dcdd780d P 2dc6ec0018f04fdfb9ae25cda2ab9650: stopping tablet replica
I20260619 16:47:57.283746 19131 raft_consensus.cc:2229] T 93a1361aecb142b88cf7e500dcdd780d P 2dc6ec0018f04fdfb9ae25cda2ab9650 [term 1 LEADER]: Raft consensus shutting down.
I20260619 16:47:57.286183 19131 raft_consensus.cc:2258] T 93a1361aecb142b88cf7e500dcdd780d P 2dc6ec0018f04fdfb9ae25cda2ab9650 [term 1 FOLLOWER]: Raft consensus is shut down!
Jun 19 16:47:57 dist-test-slave-wvz7 krb5kdc[18586](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887674, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260619 16:47:57.290040 19130 ts_tablet_manager.cc:1918] T 93a1361aecb142b88cf7e500dcdd780d P 17071c094b8b4a3481f896f88d6a987e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 16:47:57.290239 19130 log.cc:1196] T 93a1361aecb142b88cf7e500dcdd780d P 17071c094b8b4a3481f896f88d6a987e: Deleting WAL directory at /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887385903558-388-0/minicluster-data/ts-0/wal/wals/93a1361aecb142b88cf7e500dcdd780d
I20260619 16:47:57.290426 19129 tablet_replica.cc:330] T 93a1361aecb142b88cf7e500dcdd780d P ba3be5462aef45668ff34dd3e7fce2c9: stopping tablet replica
I20260619 16:47:57.290761 19130 ts_tablet_manager.cc:1939] T 93a1361aecb142b88cf7e500dcdd780d P 17071c094b8b4a3481f896f88d6a987e: Deleting consensus metadata
I20260619 16:47:57.290953 19129 raft_consensus.cc:2229] T 93a1361aecb142b88cf7e500dcdd780d P ba3be5462aef45668ff34dd3e7fce2c9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 16:47:57.291504 19129 raft_consensus.cc:2258] T 93a1361aecb142b88cf7e500dcdd780d P ba3be5462aef45668ff34dd3e7fce2c9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:47:57.291723 19131 ts_tablet_manager.cc:1905] T 93a1361aecb142b88cf7e500dcdd780d P 2dc6ec0018f04fdfb9ae25cda2ab9650: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:47:57.292745 18618 catalog_manager.cc:4841] TS 17071c094b8b4a3481f896f88d6a987e (127.0.97.1:42925): tablet 93a1361aecb142b88cf7e500dcdd780d (table test-table [id=ce4726a641064bd5ae636b051b65d11a]) successfully deleted
I20260619 16:47:57.293637 19129 ts_tablet_manager.cc:1905] T 93a1361aecb142b88cf7e500dcdd780d P ba3be5462aef45668ff34dd3e7fce2c9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:47:57.298609 19102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:47:57.269310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59918 (local address 127.0.97.1:42925)
0619 16:47:57.269904 (+ 594us) server_negotiation.cc:206] Beginning negotiation
0619 16:47:57.269910 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:47:57.275861 (+ 5951us) server_negotiation.cc:407] Connection header received
0619 16:47:57.276231 (+ 370us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:47:57.276238 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:47:57.276348 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:47:57.276425 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:47:57.278189 (+ 1764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:47:57.279376 (+ 1187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:47:57.284031 (+ 4655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:47:57.284645 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:47:57.284755 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:47:57.291208 (+ 6453us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:47:57.291223 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:47:57.291241 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:47:57.291283 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:47:57.295132 (+ 3849us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:47:57.295981 (+ 849us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:47:57.295993 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:47:57.295997 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:47:57.296153 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:47:57.296655 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:47:57.296660 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:47:57.296663 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:47:57.297136 (+ 473us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:47:57.297414 (+ 278us) server_negotiation.cc:1035] Waiting for connection context
0619 16:47:57.298017 (+ 603us) server_negotiation.cc:299] Negotiation successful
0619 16:47:57.298247 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":458}
I20260619 16:47:57.299655 19129 ts_tablet_manager.cc:1918] T 93a1361aecb142b88cf7e500dcdd780d P ba3be5462aef45668ff34dd3e7fce2c9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 16:47:57.299819 19129 log.cc:1196] T 93a1361aecb142b88cf7e500dcdd780d P ba3be5462aef45668ff34dd3e7fce2c9: Deleting WAL directory at /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887385903558-388-0/minicluster-data/ts-2/wal/wals/93a1361aecb142b88cf7e500dcdd780d
I20260619 16:47:57.300196 18815 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:47:57.300235 19129 ts_tablet_manager.cc:1939] T 93a1361aecb142b88cf7e500dcdd780d P ba3be5462aef45668ff34dd3e7fce2c9: Deleting consensus metadata
I20260619 16:47:57.302078 18618 catalog_manager.cc:4841] TS ba3be5462aef45668ff34dd3e7fce2c9 (127.0.97.3:41543): tablet 93a1361aecb142b88cf7e500dcdd780d (table test-table [id=ce4726a641064bd5ae636b051b65d11a]) successfully deleted
I20260619 16:47:57.306507 19131 ts_tablet_manager.cc:1918] T 93a1361aecb142b88cf7e500dcdd780d P 2dc6ec0018f04fdfb9ae25cda2ab9650: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 16:47:57.306653 19131 log.cc:1196] T 93a1361aecb142b88cf7e500dcdd780d P 2dc6ec0018f04fdfb9ae25cda2ab9650: Deleting WAL directory at /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887385903558-388-0/minicluster-data/ts-1/wal/wals/93a1361aecb142b88cf7e500dcdd780d
I20260619 16:47:57.307207 19131 ts_tablet_manager.cc:1939] T 93a1361aecb142b88cf7e500dcdd780d P 2dc6ec0018f04fdfb9ae25cda2ab9650: Deleting consensus metadata
I20260619 16:47:57.308946 18618 catalog_manager.cc:4841] TS 2dc6ec0018f04fdfb9ae25cda2ab9650 (127.0.97.2:46745): tablet 93a1361aecb142b88cf7e500dcdd780d (table test-table [id=ce4726a641064bd5ae636b051b65d11a]) successfully deleted
I20260619 16:47:57.393198 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu with pid 18692
I20260619 16:47:57.407068 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu pid 18826
Jun 19 16:47:57 dist-test-slave-wvz7 krb5kdc[18586](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887674, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260619 16:47:57.423930 19103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:47:57.408159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49690 (local address 127.0.97.2:46745)
0619 16:47:57.408360 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0619 16:47:57.408370 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 16:47:57.408391 (+ 21us) server_negotiation.cc:407] Connection header received
0619 16:47:57.408463 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:47:57.408468 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:47:57.408558 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:47:57.408674 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:47:57.409911 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:47:57.411068 (+ 1157us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:47:57.412320 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:47:57.413003 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:47:57.413131 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:47:57.417192 (+ 4061us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:47:57.417214 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:47:57.417234 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:47:57.417285 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:47:57.421105 (+ 3820us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:47:57.421898 (+ 793us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:47:57.421907 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:47:57.421911 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:47:57.422073 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:47:57.422390 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:47:57.422395 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:47:57.422398 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:47:57.422846 (+ 448us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:47:57.423185 (+ 339us) server_negotiation.cc:1035] Waiting for connection context
0619 16:47:57.423323 (+ 138us) server_negotiation.cc:299] Negotiation successful
0619 16:47:57.423545 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260619 16:47:57.424713 18949 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:47:57.535177 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu with pid 18826
I20260619 16:47:57.550122 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu pid 18959
Jun 19 16:47:57 dist-test-slave-wvz7 krb5kdc[18586](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887674, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260619 16:47:57.565274 19100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:47:57.550880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51582 (local address 127.0.97.3:41543)
0619 16:47:57.551059 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0619 16:47:57.551063 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:47:57.551082 (+ 19us) server_negotiation.cc:407] Connection header received
0619 16:47:57.551237 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:47:57.551241 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:47:57.551322 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:47:57.551411 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:47:57.552689 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:47:57.553401 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:47:57.554713 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:47:57.555152 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:47:57.555259 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:47:57.559793 (+ 4534us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:47:57.559808 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:47:57.559824 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:47:57.559860 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:47:57.562610 (+ 2750us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:47:57.563309 (+ 699us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:47:57.563317 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:47:57.563319 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:47:57.563498 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:47:57.563880 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:47:57.563884 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:47:57.563886 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:47:57.564239 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:47:57.564482 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0619 16:47:57.564800 (+ 318us) server_negotiation.cc:299] Negotiation successful
0619 16:47:57.564977 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260619 16:47:57.566195 19082 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:47:57.652828 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu with pid 18959
I20260619 16:47:57.665535 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu pid 18599
I20260619 16:47:57.666700 18661 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:47:57.775740 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu with pid 18599
2026-06-19T16:47:57Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (3976 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-06-19T16:47:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T16:47:57Z Disabled control of system clock
I20260619 16:47:57.843662 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu
/tmp/dist-test-taskEs2EH8/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781887385903558-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781887385903558-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781887385903558-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781887385903558-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:36389
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:33955
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:36389
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 16:47:58.157593 19140 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:47:58.157902 19140 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:47:58.157991 19140 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:47:58.165228 19140 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:47:58.165339 19140 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:47:58.165377 19140 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:47:58.165431 19140 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:47:58.165480 19140 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:47:58.174556 19140 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33955
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781887385903558-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781887385903558-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:36389
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:36389
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781887385903558-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781887385903558-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:47:58.175773 19140 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:47:58.177450 19140 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:47:58.188694 19145 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:47:58.188665 19146 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:47:58.188771 19148 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:47:58.189369 19140 server_base.cc:1034] running on GCE node
I20260619 16:47:58.190171 19140 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:47:58.191603 19140 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:47:58.192818 19140 hybrid_clock.cc:648] HybridClock initialized: now 1781887678192763 us; error 46 us; skew 500 ppm
I20260619 16:47:58.193153 19140 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 16:47:58.195385 19140 webserver.cc:460] Webserver started at http://127.0.97.62:34115/ using document root <none> and password file <none>
I20260619 16:47:58.196065 19140 fs_manager.cc:359] Metadata directory not provided
I20260619 16:47:58.196163 19140 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:47:58.196465 19140 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:47:58.199334 19140 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781887385903558-388-0/minicluster-data/master-0/data/instance:
uuid: "90d117b5d0eb4a4687558819b254295c"
format_stamp: "Formatted at 2026-06-19 16:47:58 on dist-test-slave-wvz7"
server_key: "aa7baf8046720ebc410f516809d5ae4f"
server_key_iv: "e788e8bfb486c1c7c11f3315e9fe8445"
server_key_version: "encryptionkey@0"
I20260619 16:47:58.200105 19140 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781887385903558-388-0/minicluster-data/master-0/wal/instance:
uuid: "90d117b5d0eb4a4687558819b254295c"
format_stamp: "Formatted at 2026-06-19 16:47:58 on dist-test-slave-wvz7"
server_key: "aa7baf8046720ebc410f516809d5ae4f"
server_key_iv: "e788e8bfb486c1c7c11f3315e9fe8445"
server_key_version: "encryptionkey@0"
I20260619 16:47:58.205089 19140 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.003s
I20260619 16:47:58.208422 19154 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:47:58.209761 19140 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260619 16:47:58.209924 19140 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781887385903558-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781887385903558-388-0/minicluster-data/master-0/wal
uuid: "90d117b5d0eb4a4687558819b254295c"
format_stamp: "Formatted at 2026-06-19 16:47:58 on dist-test-slave-wvz7"
server_key: "aa7baf8046720ebc410f516809d5ae4f"
server_key_iv: "e788e8bfb486c1c7c11f3315e9fe8445"
server_key_version: "encryptionkey@0"
I20260619 16:47:58.210114 19140 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781887385903558-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781887385903558-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781887385903558-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:47:58.239578 19140 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:47:58.240423 19140 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:47:58.240689 19140 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:47:58.257467 19140 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:36389
I20260619 16:47:58.257481 19205 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:36389 every 8 connection(s)
I20260619 16:47:58.258780 19140 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781887385903558-388-0/minicluster-data/master-0/data/info.pb
I20260619 16:47:58.262542 19206 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:47:58.264278 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu as pid 19140
I20260619 16:47:58.264456 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781887385903558-388-0/minicluster-data/master-0/wal/instance
I20260619 16:47:58.264966 388 external_mini_cluster.cc:1428] Setting key 805185aa6c5824966b257b4223ff8465
I20260619 16:47:58.277869 19206 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 90d117b5d0eb4a4687558819b254295c: Bootstrap starting.
I20260619 16:47:58.281682 19206 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 90d117b5d0eb4a4687558819b254295c: Neither blocks nor log segments found. Creating new log.
I20260619 16:47:58.282394 19209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:47:58.266992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33596 (local address 127.0.97.62:36389)
0619 16:47:58.269991 (+ 2999us) server_negotiation.cc:206] Beginning negotiation
0619 16:47:58.270007 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0619 16:47:58.270059 (+ 52us) server_negotiation.cc:407] Connection header received
0619 16:47:58.272312 (+ 2253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:47:58.272341 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:47:58.272937 (+ 596us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:47:58.273351 (+ 414us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:47:58.273927 (+ 576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:47:58.275915 (+ 1988us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:47:58.277238 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:47:58.278264 (+ 1026us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:47:58.278457 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:47:58.279613 (+ 1156us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:47:58.279623 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:47:58.279640 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:47:58.280334 (+ 694us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:47:58.280444 (+ 110us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:47:58.280479 (+ 35us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:47:58.280591 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0619 16:47:58.280814 (+ 223us) server_negotiation.cc:299] Negotiation successful
0619 16:47:58.281117 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2438,"thread_start_us":2215,"threads_started":1}
I20260619 16:47:58.282922 19206 log.cc:824] T 00000000000000000000000000000000 P 90d117b5d0eb4a4687558819b254295c: Log is configured to *not* fsync() on all Append() calls
I20260619 16:47:58.285876 19206 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 90d117b5d0eb4a4687558819b254295c: No bootstrap required, opened a new log
I20260619 16:47:58.293145 19206 raft_consensus.cc:348] T 00000000000000000000000000000000 P 90d117b5d0eb4a4687558819b254295c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90d117b5d0eb4a4687558819b254295c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36389 } }
I20260619 16:47:58.293493 19206 raft_consensus.cc:374] T 00000000000000000000000000000000 P 90d117b5d0eb4a4687558819b254295c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:47:58.293582 19206 raft_consensus.cc:729] T 00000000000000000000000000000000 P 90d117b5d0eb4a4687558819b254295c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90d117b5d0eb4a4687558819b254295c, State: Initialized, Role: FOLLOWER
I20260619 16:47:58.294134 19206 consensus_queue.cc:260] T 00000000000000000000000000000000 P 90d117b5d0eb4a4687558819b254295c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90d117b5d0eb4a4687558819b254295c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36389 } }
I20260619 16:47:58.294348 19206 raft_consensus.cc:388] T 00000000000000000000000000000000 P 90d117b5d0eb4a4687558819b254295c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:47:58.294464 19206 raft_consensus.cc:482] T 00000000000000000000000000000000 P 90d117b5d0eb4a4687558819b254295c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:47:58.294615 19206 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 90d117b5d0eb4a4687558819b254295c [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:47:58.297148 19206 raft_consensus.cc:504] T 00000000000000000000000000000000 P 90d117b5d0eb4a4687558819b254295c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90d117b5d0eb4a4687558819b254295c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36389 } }
I20260619 16:47:58.297639 19206 leader_election.cc:302] T 00000000000000000000000000000000 P 90d117b5d0eb4a4687558819b254295c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 90d117b5d0eb4a4687558819b254295c; no voters:
I20260619 16:47:58.298242 19206 leader_election.cc:288] T 00000000000000000000000000000000 P 90d117b5d0eb4a4687558819b254295c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 16:47:58.298427 19211 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 90d117b5d0eb4a4687558819b254295c [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:47:58.299120 19211 raft_consensus.cc:686] T 00000000000000000000000000000000 P 90d117b5d0eb4a4687558819b254295c [term 1 LEADER]: Becoming Leader. State: Replica: 90d117b5d0eb4a4687558819b254295c, State: Running, Role: LEADER
I20260619 16:47:58.299657 19211 consensus_queue.cc:237] T 00000000000000000000000000000000 P 90d117b5d0eb4a4687558819b254295c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90d117b5d0eb4a4687558819b254295c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36389 } }
I20260619 16:47:58.300163 19206 sys_catalog.cc:564] T 00000000000000000000000000000000 P 90d117b5d0eb4a4687558819b254295c [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:47:58.302456 19212 sys_catalog.cc:455] T 00000000000000000000000000000000 P 90d117b5d0eb4a4687558819b254295c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "90d117b5d0eb4a4687558819b254295c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90d117b5d0eb4a4687558819b254295c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36389 } } }
I20260619 16:47:58.302891 19212 sys_catalog.cc:458] T 00000000000000000000000000000000 P 90d117b5d0eb4a4687558819b254295c [sys.catalog]: This master's current role is: LEADER
I20260619 16:47:58.302430 19213 sys_catalog.cc:455] T 00000000000000000000000000000000 P 90d117b5d0eb4a4687558819b254295c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 90d117b5d0eb4a4687558819b254295c. Latest consensus state: current_term: 1 leader_uuid: "90d117b5d0eb4a4687558819b254295c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90d117b5d0eb4a4687558819b254295c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36389 } } }
I20260619 16:47:58.303508 19213 sys_catalog.cc:458] T 00000000000000000000000000000000 P 90d117b5d0eb4a4687558819b254295c [sys.catalog]: This master's current role is: LEADER
I20260619 16:47:58.305074 19217 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:47:58.310429 19217 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:47:58.319785 19217 catalog_manager.cc:1347] Generated new cluster ID: 1bde01ebff794d2b9cdbde77e88c0a28
I20260619 16:47:58.319900 19217 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:47:58.330677 19217 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 16:47:58.331960 19217 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:47:58.339023 19217 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 90d117b5d0eb4a4687558819b254295c: Generated new TSK 0
I20260619 16:47:58.339893 19217 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 16:47:58.354773 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu
/tmp/dist-test-taskEs2EH8/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781887385903558-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781887385903558-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781887385903558-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781887385903558-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36389
--builtin_ntp_servers=127.0.97.20:33955
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 16:47:58.672011 19230 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:47:58.672291 19230 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:47:58.672405 19230 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:47:58.679582 19230 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:47:58.679684 19230 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:47:58.679824 19230 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260619 16:47:58.689083 19230 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33955
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781887385903558-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781887385903558-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781887385903558-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36389
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781887385903558-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:47:58.690371 19230 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:47:58.692072 19230 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:47:58.704239 19235 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:47:58.704165 19236 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:47:58.704195 19238 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:47:58.704861 19230 server_base.cc:1034] running on GCE node
I20260619 16:47:58.705502 19230 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:47:58.706482 19230 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:47:58.707690 19230 hybrid_clock.cc:648] HybridClock initialized: now 1781887678707627 us; error 50 us; skew 500 ppm
I20260619 16:47:58.708007 19230 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 16:47:58.710253 19230 webserver.cc:460] Webserver started at http://127.0.97.1:36859/ using document root <none> and password file <none>
I20260619 16:47:58.710934 19230 fs_manager.cc:359] Metadata directory not provided
I20260619 16:47:58.711033 19230 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:47:58.711333 19230 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:47:58.714207 19230 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781887385903558-388-0/minicluster-data/ts-0/data/instance:
uuid: "69975283742140389a2325c7b88e814b"
format_stamp: "Formatted at 2026-06-19 16:47:58 on dist-test-slave-wvz7"
server_key: "391e731af2d2546110783fac7039d316"
server_key_iv: "1efec706a019279ff8ff8a73aec4be59"
server_key_version: "encryptionkey@0"
I20260619 16:47:58.714951 19230 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781887385903558-388-0/minicluster-data/ts-0/wal/instance:
uuid: "69975283742140389a2325c7b88e814b"
format_stamp: "Formatted at 2026-06-19 16:47:58 on dist-test-slave-wvz7"
server_key: "391e731af2d2546110783fac7039d316"
server_key_iv: "1efec706a019279ff8ff8a73aec4be59"
server_key_version: "encryptionkey@0"
I20260619 16:47:58.719866 19230 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260619 16:47:58.723222 19244 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:47:58.724534 19230 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 16:47:58.724694 19230 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781887385903558-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781887385903558-388-0/minicluster-data/ts-0/wal
uuid: "69975283742140389a2325c7b88e814b"
format_stamp: "Formatted at 2026-06-19 16:47:58 on dist-test-slave-wvz7"
server_key: "391e731af2d2546110783fac7039d316"
server_key_iv: "1efec706a019279ff8ff8a73aec4be59"
server_key_version: "encryptionkey@0"
I20260619 16:47:58.724902 19230 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781887385903558-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781887385903558-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781887385903558-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:47:58.748172 19230 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:47:58.748967 19230 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:47:58.749226 19230 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:47:58.750214 19230 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:47:58.751645 19230 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:47:58.751729 19230 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:47:58.751830 19230 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:47:58.751904 19230 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:47:58.780226 19230 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:37667
I20260619 16:47:58.780278 19356 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:37667 every 8 connection(s)
I20260619 16:47:58.781469 19230 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781887385903558-388-0/minicluster-data/ts-0/data/info.pb
I20260619 16:47:58.790026 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu as pid 19230
I20260619 16:47:58.790181 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781887385903558-388-0/minicluster-data/ts-0/wal/instance
I20260619 16:47:58.790607 388 external_mini_cluster.cc:1428] Setting key 13345930d8f87e4b3a5215865a13f93c
I20260619 16:47:58.793165 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu
/tmp/dist-test-taskEs2EH8/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781887385903558-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781887385903558-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781887385903558-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781887385903558-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36389
--builtin_ntp_servers=127.0.97.20:33955
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260619 16:47:58.794049 19359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:47:58.784022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:33365 (local address 127.0.97.62:36389)
0619 16:47:58.786099 (+ 2077us) server_negotiation.cc:206] Beginning negotiation
0619 16:47:58.786105 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:47:58.786126 (+ 21us) server_negotiation.cc:407] Connection header received
0619 16:47:58.787782 (+ 1656us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:47:58.787788 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:47:58.787873 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:47:58.787966 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:47:58.789035 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:47:58.789728 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:47:58.791312 (+ 1584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:47:58.791743 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:47:58.791856 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:47:58.793160 (+ 1304us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:47:58.793165 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:47:58.793169 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:47:58.793186 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:47:58.793232 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:47:58.793243 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:47:58.793314 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0619 16:47:58.793566 (+ 252us) server_negotiation.cc:299] Negotiation successful
0619 16:47:58.793661 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1946,"thread_start_us":1839,"threads_started":1}
I20260619 16:47:58.795013 19360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:47:58.784739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36389 (local address 127.0.97.1:33365)
0619 16:47:58.785783 (+ 1044us) negotiation.cc:107] Waiting for socket to connect
0619 16:47:58.785842 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0619 16:47:58.787520 (+ 1678us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:47:58.788216 (+ 696us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:47:58.788225 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:47:58.788331 (+ 106us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:47:58.788878 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:47:58.788894 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:47:58.789897 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:47:58.789900 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:47:58.791180 (+ 1280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:47:58.791192 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:47:58.791955 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:47:58.791962 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:47:58.792182 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:47:58.792851 (+ 669us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 16:47:58.792853 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:47:58.792898 (+ 45us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 16:47:58.792979 (+ 81us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:47:58.793362 (+ 383us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:47:58.793373 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:47:58.793381 (+ 8us) client_negotiation.cc:769] Sending connection context
0619 16:47:58.793542 (+ 161us) client_negotiation.cc:240] Negotiation successful
0619 16:47:58.793970 (+ 428us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":566,"thread_start_us":330,"threads_started":1}
I20260619 16:47:58.796619 19357 heartbeater.cc:347] Connected to a master server at 127.0.97.62:36389
I20260619 16:47:58.796914 19357 heartbeater.cc:464] Registering TS with master...
I20260619 16:47:58.797610 19357 heartbeater.cc:511] Master 127.0.97.62:36389 requested a full tablet report, sending...
I20260619 16:47:58.799710 19171 ts_manager.cc:194] Registered new tserver with Master: 69975283742140389a2325c7b88e814b (127.0.97.1:37667)
I20260619 16:47:58.801385 19171 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.1:33365
W20260619 16:47:59.107443 19361 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:47:59.107728 19361 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:47:59.107854 19361 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:47:59.115216 19361 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:47:59.115366 19361 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:47:59.115516 19361 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260619 16:47:59.124832 19361 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33955
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781887385903558-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781887385903558-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781887385903558-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36389
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781887385903558-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:47:59.126147 19361 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:47:59.127905 19361 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:47:59.140151 19366 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:47:59.140126 19369 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:47:59.140095 19367 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:47:59.140947 19361 server_base.cc:1034] running on GCE node
I20260619 16:47:59.141566 19361 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:47:59.142518 19361 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:47:59.143743 19361 hybrid_clock.cc:648] HybridClock initialized: now 1781887679143691 us; error 44 us; skew 500 ppm
I20260619 16:47:59.144073 19361 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 16:47:59.146427 19361 webserver.cc:460] Webserver started at http://127.0.97.2:42537/ using document root <none> and password file <none>
I20260619 16:47:59.147091 19361 fs_manager.cc:359] Metadata directory not provided
I20260619 16:47:59.147188 19361 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:47:59.147496 19361 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:47:59.150308 19361 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781887385903558-388-0/minicluster-data/ts-1/data/instance:
uuid: "bae81e16ae254fb9b3b69fc6fe0c515b"
format_stamp: "Formatted at 2026-06-19 16:47:59 on dist-test-slave-wvz7"
server_key: "a35ff95468fe466c3227040831d5e97e"
server_key_iv: "16efaff654dea6b7ece24a2279ca624b"
server_key_version: "encryptionkey@0"
I20260619 16:47:59.151103 19361 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781887385903558-388-0/minicluster-data/ts-1/wal/instance:
uuid: "bae81e16ae254fb9b3b69fc6fe0c515b"
format_stamp: "Formatted at 2026-06-19 16:47:59 on dist-test-slave-wvz7"
server_key: "a35ff95468fe466c3227040831d5e97e"
server_key_iv: "16efaff654dea6b7ece24a2279ca624b"
server_key_version: "encryptionkey@0"
I20260619 16:47:59.155997 19361 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260619 16:47:59.159339 19375 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:47:59.160621 19361 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 16:47:59.160764 19361 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781887385903558-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781887385903558-388-0/minicluster-data/ts-1/wal
uuid: "bae81e16ae254fb9b3b69fc6fe0c515b"
format_stamp: "Formatted at 2026-06-19 16:47:59 on dist-test-slave-wvz7"
server_key: "a35ff95468fe466c3227040831d5e97e"
server_key_iv: "16efaff654dea6b7ece24a2279ca624b"
server_key_version: "encryptionkey@0"
I20260619 16:47:59.160956 19361 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781887385903558-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781887385903558-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781887385903558-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:47:59.198582 19361 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:47:59.199447 19361 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:47:59.199702 19361 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:47:59.200762 19361 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:47:59.202332 19361 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:47:59.202435 19361 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:47:59.202536 19361 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:47:59.202601 19361 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:47:59.232432 19361 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:37867
I20260619 16:47:59.232470 19487 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:37867 every 8 connection(s)
I20260619 16:47:59.233669 19361 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781887385903558-388-0/minicluster-data/ts-1/data/info.pb
I20260619 16:47:59.237882 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu as pid 19361
I20260619 16:47:59.238044 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781887385903558-388-0/minicluster-data/ts-1/wal/instance
I20260619 16:47:59.238518 388 external_mini_cluster.cc:1428] Setting key 8975d37e42d46c46180d2e221bffc354
I20260619 16:47:59.241576 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu
/tmp/dist-test-taskEs2EH8/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781887385903558-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781887385903558-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781887385903558-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781887385903558-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36389
--builtin_ntp_servers=127.0.97.20:33955
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260619 16:47:59.247385 19359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:47:59.236167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:42771 (local address 127.0.97.62:36389)
0619 16:47:59.236381 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0619 16:47:59.236388 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:47:59.238361 (+ 1973us) server_negotiation.cc:407] Connection header received
0619 16:47:59.240706 (+ 2345us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:47:59.240711 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:47:59.240793 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:47:59.240898 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:47:59.242046 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:47:59.243073 (+ 1027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:47:59.244689 (+ 1616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:47:59.245107 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:47:59.245219 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:47:59.246606 (+ 1387us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:47:59.246612 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:47:59.246617 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:47:59.246636 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:47:59.246669 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:47:59.246680 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:47:59.246754 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0619 16:47:59.246971 (+ 217us) server_negotiation.cc:299] Negotiation successful
0619 16:47:59.247069 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260619 16:47:59.248409 19490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:47:59.236829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36389 (local address 127.0.97.2:42771)
0619 16:47:59.238158 (+ 1329us) negotiation.cc:107] Waiting for socket to connect
0619 16:47:59.238217 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0619 16:47:59.240383 (+ 2166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:47:59.241085 (+ 702us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:47:59.241094 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:47:59.241191 (+ 97us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:47:59.241884 (+ 693us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:47:59.241903 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:47:59.243236 (+ 1333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:47:59.243240 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:47:59.244540 (+ 1300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:47:59.244551 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:47:59.245325 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:47:59.245332 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:47:59.245586 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:47:59.246271 (+ 685us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 16:47:59.246273 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:47:59.246312 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 16:47:59.246430 (+ 118us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:47:59.246763 (+ 333us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:47:59.246774 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:47:59.246782 (+ 8us) client_negotiation.cc:769] Sending connection context
0619 16:47:59.246935 (+ 153us) client_negotiation.cc:240] Negotiation successful
0619 16:47:59.247345 (+ 410us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":661,"thread_start_us":353,"threads_started":1}
I20260619 16:47:59.249883 19488 heartbeater.cc:347] Connected to a master server at 127.0.97.62:36389
I20260619 16:47:59.250208 19488 heartbeater.cc:464] Registering TS with master...
I20260619 16:47:59.251068 19488 heartbeater.cc:511] Master 127.0.97.62:36389 requested a full tablet report, sending...
I20260619 16:47:59.252592 19171 ts_manager.cc:194] Registered new tserver with Master: bae81e16ae254fb9b3b69fc6fe0c515b (127.0.97.2:37867)
I20260619 16:47:59.253386 19171 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.2:42771
W20260619 16:47:59.556140 19491 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:47:59.556428 19491 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:47:59.556555 19491 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:47:59.563753 19491 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:47:59.563863 19491 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:47:59.564034 19491 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260619 16:47:59.573177 19491 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33955
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781887385903558-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781887385903558-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781887385903558-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36389
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781887385903558-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:47:59.574540 19491 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:47:59.576262 19491 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:47:59.588539 19496 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:47:59.588562 19499 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:47:59.588595 19497 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:47:59.589298 19491 server_base.cc:1034] running on GCE node
I20260619 16:47:59.589916 19491 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:47:59.590806 19491 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:47:59.592031 19491 hybrid_clock.cc:648] HybridClock initialized: now 1781887679591975 us; error 47 us; skew 500 ppm
I20260619 16:47:59.592335 19491 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 16:47:59.594475 19491 webserver.cc:460] Webserver started at http://127.0.97.3:44689/ using document root <none> and password file <none>
I20260619 16:47:59.595139 19491 fs_manager.cc:359] Metadata directory not provided
I20260619 16:47:59.595238 19491 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:47:59.595528 19491 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:47:59.598259 19491 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781887385903558-388-0/minicluster-data/ts-2/data/instance:
uuid: "214cb7fd381a4288bb71cd4e11baee5c"
format_stamp: "Formatted at 2026-06-19 16:47:59 on dist-test-slave-wvz7"
server_key: "309529599f2748799e33c539e14e8b13"
server_key_iv: "522c5fc86547f4d9bcda39b2b96416a5"
server_key_version: "encryptionkey@0"
I20260619 16:47:59.598979 19491 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781887385903558-388-0/minicluster-data/ts-2/wal/instance:
uuid: "214cb7fd381a4288bb71cd4e11baee5c"
format_stamp: "Formatted at 2026-06-19 16:47:59 on dist-test-slave-wvz7"
server_key: "309529599f2748799e33c539e14e8b13"
server_key_iv: "522c5fc86547f4d9bcda39b2b96416a5"
server_key_version: "encryptionkey@0"
I20260619 16:47:59.603744 19491 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 16:47:59.606951 19505 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:47:59.608244 19491 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 16:47:59.608383 19491 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781887385903558-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781887385903558-388-0/minicluster-data/ts-2/wal
uuid: "214cb7fd381a4288bb71cd4e11baee5c"
format_stamp: "Formatted at 2026-06-19 16:47:59 on dist-test-slave-wvz7"
server_key: "309529599f2748799e33c539e14e8b13"
server_key_iv: "522c5fc86547f4d9bcda39b2b96416a5"
server_key_version: "encryptionkey@0"
I20260619 16:47:59.608566 19491 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781887385903558-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781887385903558-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781887385903558-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:47:59.626899 19491 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:47:59.627727 19491 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:47:59.627982 19491 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:47:59.629024 19491 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:47:59.630553 19491 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:47:59.630649 19491 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:47:59.630751 19491 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:47:59.630817 19491 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:47:59.660329 19491 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:45361
I20260619 16:47:59.660377 19617 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:45361 every 8 connection(s)
I20260619 16:47:59.661573 19491 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781887385903558-388-0/minicluster-data/ts-2/data/info.pb
I20260619 16:47:59.666054 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu as pid 19491
I20260619 16:47:59.666190 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781887385903558-388-0/minicluster-data/ts-2/wal/instance
I20260619 16:47:59.666584 388 external_mini_cluster.cc:1428] Setting key 1abf0373b50d6253b419ef13cb64a139
I20260619 16:47:59.673491 19359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:47:59.664031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:35391 (local address 127.0.97.62:36389)
0619 16:47:59.664257 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0619 16:47:59.664264 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:47:59.665678 (+ 1414us) server_negotiation.cc:407] Connection header received
0619 16:47:59.667596 (+ 1918us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:47:59.667600 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:47:59.667676 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:47:59.667781 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:47:59.668990 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:47:59.669813 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:47:59.671252 (+ 1439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:47:59.671648 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:47:59.671736 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:47:59.672883 (+ 1147us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:47:59.672887 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:47:59.672891 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:47:59.672908 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:47:59.672941 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:47:59.672952 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:47:59.673027 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0619 16:47:59.673177 (+ 150us) server_negotiation.cc:299] Negotiation successful
0619 16:47:59.673252 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260619 16:47:59.674463 19620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:47:59.664540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36389 (local address 127.0.97.3:35391)
0619 16:47:59.665539 (+ 999us) negotiation.cc:107] Waiting for socket to connect
0619 16:47:59.665574 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0619 16:47:59.667331 (+ 1757us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:47:59.668043 (+ 712us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:47:59.668054 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:47:59.668167 (+ 113us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:47:59.668813 (+ 646us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:47:59.668830 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:47:59.669949 (+ 1119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:47:59.669952 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:47:59.671145 (+ 1193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:47:59.671157 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:47:59.671802 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:47:59.671806 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:47:59.672013 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:47:59.672609 (+ 596us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 16:47:59.672611 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:47:59.672648 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 16:47:59.672725 (+ 77us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:47:59.673030 (+ 305us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:47:59.673036 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:47:59.673041 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 16:47:59.673164 (+ 123us) client_negotiation.cc:240] Negotiation successful
0619 16:47:59.673450 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":520,"thread_start_us":269,"threads_started":1}
I20260619 16:47:59.675806 19618 heartbeater.cc:347] Connected to a master server at 127.0.97.62:36389
I20260619 16:47:59.676100 19618 heartbeater.cc:464] Registering TS with master...
I20260619 16:47:59.676807 19618 heartbeater.cc:511] Master 127.0.97.62:36389 requested a full tablet report, sending...
I20260619 16:47:59.678143 19171 ts_manager.cc:194] Registered new tserver with Master: 214cb7fd381a4288bb71cd4e11baee5c (127.0.97.3:45361)
I20260619 16:47:59.678942 19171 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.3:35391
I20260619 16:47:59.681371 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 16:47:59.690575 19359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:47:59.685197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33598 (local address 127.0.97.62:36389)
0619 16:47:59.685408 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0619 16:47:59.685414 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:47:59.685718 (+ 304us) server_negotiation.cc:407] Connection header received
0619 16:47:59.685885 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:47:59.685888 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:47:59.685971 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:47:59.686048 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:47:59.686444 (+ 396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:47:59.687108 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:47:59.688398 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:47:59.688774 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:47:59.688865 (+ 91us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:47:59.689851 (+ 986us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:47:59.689855 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:47:59.689859 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:47:59.689876 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:47:59.689920 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:47:59.689932 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:47:59.690038 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0619 16:47:59.690232 (+ 194us) server_negotiation.cc:299] Negotiation successful
0619 16:47:59.690289 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260619 16:47:59.693954 19171 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33598:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 16:47:59.700596 19171 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 16:47:59.737049 19632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:47:59.712411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51654 (local address 127.0.97.1:37667)
0619 16:47:59.726175 (+ 13764us) server_negotiation.cc:206] Beginning negotiation
0619 16:47:59.726186 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 16:47:59.726219 (+ 33us) server_negotiation.cc:407] Connection header received
0619 16:47:59.726318 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:47:59.726325 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:47:59.726618 (+ 293us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:47:59.726808 (+ 190us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:47:59.728298 (+ 1490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:47:59.730005 (+ 1707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:47:59.734326 (+ 4321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:47:59.735555 (+ 1229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:47:59.735676 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:47:59.735931 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0619 16:47:59.736528 (+ 597us) server_negotiation.cc:299] Negotiation successful
0619 16:47:59.736693 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13608,"thread_start_us":13493,"threads_started":1}
I20260619 16:47:59.737174 19630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:47:59.712143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:37667 (local address 127.0.0.1:51654)
0619 16:47:59.722837 (+ 10694us) negotiation.cc:107] Waiting for socket to connect
0619 16:47:59.722899 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0619 16:47:59.723103 (+ 204us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:47:59.726830 (+ 3727us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:47:59.726838 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:47:59.726866 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:47:59.727219 (+ 353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:47:59.727233 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:47:59.730164 (+ 2931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:47:59.730170 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:47:59.734127 (+ 3957us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:47:59.734143 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:47:59.736060 (+ 1917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:47:59.736067 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:47:59.736279 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:47:59.736343 (+ 64us) client_negotiation.cc:769] Sending connection context
0619 16:47:59.736468 (+ 125us) client_negotiation.cc:240] Negotiation successful
0619 16:47:59.736780 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10360,"mutex_wait_us":180,"thread_start_us":11169,"threads_started":1}
I20260619 16:47:59.737304 19631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:47:59.712327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:37867 (local address 127.0.0.1:36356)
0619 16:47:59.726500 (+ 14173us) negotiation.cc:107] Waiting for socket to connect
0619 16:47:59.726522 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0619 16:47:59.726626 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:47:59.728999 (+ 2373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:47:59.729005 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:47:59.729035 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:47:59.729452 (+ 417us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:47:59.729465 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:47:59.732334 (+ 2869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:47:59.732343 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:47:59.734373 (+ 2030us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:47:59.734385 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:47:59.736520 (+ 2135us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:47:59.736527 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:47:59.736708 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:47:59.736741 (+ 33us) client_negotiation.cc:769] Sending connection context
0619 16:47:59.736820 (+ 79us) client_negotiation.cc:240] Negotiation successful
0619 16:47:59.736946 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14055,"mutex_wait_us":22,"thread_start_us":13930,"threads_started":1}
I20260619 16:47:59.738099 19633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:47:59.712991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36356 (local address 127.0.97.2:37867)
0619 16:47:59.728135 (+ 15144us) server_negotiation.cc:206] Beginning negotiation
0619 16:47:59.728146 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 16:47:59.728176 (+ 30us) server_negotiation.cc:407] Connection header received
0619 16:47:59.728286 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:47:59.728294 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:47:59.728554 (+ 260us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:47:59.729673 (+ 1119us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:47:59.729978 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:47:59.731634 (+ 1656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:47:59.734552 (+ 2918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:47:59.735996 (+ 1444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:47:59.736113 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:47:59.736380 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0619 16:47:59.737549 (+ 1169us) server_negotiation.cc:299] Negotiation successful
0619 16:47:59.737723 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14961,"thread_start_us":15714,"threads_started":1}
I20260619 16:47:59.741264 19423 tablet_service.cc:1467] Processing CreateTablet for tablet defd28bf1f3844329ed93a775d69e05b (DEFAULT_TABLE table=test-table [id=7eda522923db41159d1432f4985b5a4d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:47:59.741789 19635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:47:59.727486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:45361 (local address 127.0.0.1:37706)
0619 16:47:59.732714 (+ 5228us) negotiation.cc:107] Waiting for socket to connect
0619 16:47:59.732734 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 16:47:59.732883 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:47:59.733463 (+ 580us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:47:59.733468 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:47:59.733493 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:47:59.733941 (+ 448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:47:59.733954 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:47:59.736160 (+ 2206us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:47:59.736165 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:47:59.738688 (+ 2523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:47:59.738702 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:47:59.741053 (+ 2351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:47:59.741059 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:47:59.741230 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:47:59.741262 (+ 32us) client_negotiation.cc:769] Sending connection context
0619 16:47:59.741352 (+ 90us) client_negotiation.cc:240] Negotiation successful
0619 16:47:59.741476 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5098,"thread_start_us":4945,"threads_started":1}
I20260619 16:47:59.741241 19292 tablet_service.cc:1467] Processing CreateTablet for tablet defd28bf1f3844329ed93a775d69e05b (DEFAULT_TABLE table=test-table [id=7eda522923db41159d1432f4985b5a4d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:47:59.742527 19634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:47:59.727038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37706 (local address 127.0.97.3:45361)
0619 16:47:59.732202 (+ 5164us) server_negotiation.cc:206] Beginning negotiation
0619 16:47:59.732208 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:47:59.732856 (+ 648us) server_negotiation.cc:407] Connection header received
0619 16:47:59.733027 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:47:59.733035 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:47:59.733269 (+ 234us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:47:59.733454 (+ 185us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:47:59.734156 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:47:59.735996 (+ 1840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:47:59.738870 (+ 2874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:47:59.740183 (+ 1313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:47:59.740679 (+ 496us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:47:59.740931 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0619 16:47:59.742009 (+ 1078us) server_negotiation.cc:299] Negotiation successful
0619 16:47:59.742174 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5024,"thread_start_us":4879,"threads_started":1}
I20260619 16:47:59.742550 19423 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet defd28bf1f3844329ed93a775d69e05b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:47:59.742633 19292 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet defd28bf1f3844329ed93a775d69e05b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:47:59.748696 19553 tablet_service.cc:1467] Processing CreateTablet for tablet defd28bf1f3844329ed93a775d69e05b (DEFAULT_TABLE table=test-table [id=7eda522923db41159d1432f4985b5a4d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:47:59.750010 19553 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet defd28bf1f3844329ed93a775d69e05b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:47:59.755869 19636 tablet_bootstrap.cc:492] T defd28bf1f3844329ed93a775d69e05b P 69975283742140389a2325c7b88e814b: Bootstrap starting.
I20260619 16:47:59.756635 19637 tablet_bootstrap.cc:492] T defd28bf1f3844329ed93a775d69e05b P bae81e16ae254fb9b3b69fc6fe0c515b: Bootstrap starting.
I20260619 16:47:59.759824 19636 tablet_bootstrap.cc:654] T defd28bf1f3844329ed93a775d69e05b P 69975283742140389a2325c7b88e814b: Neither blocks nor log segments found. Creating new log.
I20260619 16:47:59.759994 19637 tablet_bootstrap.cc:654] T defd28bf1f3844329ed93a775d69e05b P bae81e16ae254fb9b3b69fc6fe0c515b: Neither blocks nor log segments found. Creating new log.
I20260619 16:47:59.760839 19636 log.cc:824] T defd28bf1f3844329ed93a775d69e05b P 69975283742140389a2325c7b88e814b: Log is configured to *not* fsync() on all Append() calls
I20260619 16:47:59.761097 19637 log.cc:824] T defd28bf1f3844329ed93a775d69e05b P bae81e16ae254fb9b3b69fc6fe0c515b: Log is configured to *not* fsync() on all Append() calls
I20260619 16:47:59.763463 19640 tablet_bootstrap.cc:492] T defd28bf1f3844329ed93a775d69e05b P 214cb7fd381a4288bb71cd4e11baee5c: Bootstrap starting.
I20260619 16:47:59.764223 19637 tablet_bootstrap.cc:492] T defd28bf1f3844329ed93a775d69e05b P bae81e16ae254fb9b3b69fc6fe0c515b: No bootstrap required, opened a new log
I20260619 16:47:59.764509 19637 ts_tablet_manager.cc:1397] T defd28bf1f3844329ed93a775d69e05b P bae81e16ae254fb9b3b69fc6fe0c515b: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260619 16:47:59.764958 19636 tablet_bootstrap.cc:492] T defd28bf1f3844329ed93a775d69e05b P 69975283742140389a2325c7b88e814b: No bootstrap required, opened a new log
I20260619 16:47:59.765223 19636 ts_tablet_manager.cc:1397] T defd28bf1f3844329ed93a775d69e05b P 69975283742140389a2325c7b88e814b: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.003s
I20260619 16:47:59.767058 19640 tablet_bootstrap.cc:654] T defd28bf1f3844329ed93a775d69e05b P 214cb7fd381a4288bb71cd4e11baee5c: Neither blocks nor log segments found. Creating new log.
I20260619 16:47:59.768198 19640 log.cc:824] T defd28bf1f3844329ed93a775d69e05b P 214cb7fd381a4288bb71cd4e11baee5c: Log is configured to *not* fsync() on all Append() calls
I20260619 16:47:59.771157 19640 tablet_bootstrap.cc:492] T defd28bf1f3844329ed93a775d69e05b P 214cb7fd381a4288bb71cd4e11baee5c: No bootstrap required, opened a new log
I20260619 16:47:59.771410 19640 ts_tablet_manager.cc:1397] T defd28bf1f3844329ed93a775d69e05b P 214cb7fd381a4288bb71cd4e11baee5c: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260619 16:47:59.771750 19637 raft_consensus.cc:348] T defd28bf1f3844329ed93a775d69e05b P bae81e16ae254fb9b3b69fc6fe0c515b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69975283742140389a2325c7b88e814b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37667 } } peers { permanent_uuid: "bae81e16ae254fb9b3b69fc6fe0c515b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37867 } } peers { permanent_uuid: "214cb7fd381a4288bb71cd4e11baee5c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45361 } }
I20260619 16:47:59.772087 19637 raft_consensus.cc:374] T defd28bf1f3844329ed93a775d69e05b P bae81e16ae254fb9b3b69fc6fe0c515b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:47:59.772207 19637 raft_consensus.cc:729] T defd28bf1f3844329ed93a775d69e05b P bae81e16ae254fb9b3b69fc6fe0c515b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bae81e16ae254fb9b3b69fc6fe0c515b, State: Initialized, Role: FOLLOWER
I20260619 16:47:59.772769 19637 consensus_queue.cc:260] T defd28bf1f3844329ed93a775d69e05b P bae81e16ae254fb9b3b69fc6fe0c515b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69975283742140389a2325c7b88e814b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37667 } } peers { permanent_uuid: "bae81e16ae254fb9b3b69fc6fe0c515b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37867 } } peers { permanent_uuid: "214cb7fd381a4288bb71cd4e11baee5c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45361 } }
I20260619 16:47:59.774353 19637 ts_tablet_manager.cc:1428] T defd28bf1f3844329ed93a775d69e05b P bae81e16ae254fb9b3b69fc6fe0c515b: Time spent starting tablet: real 0.010s user 0.010s sys 0.000s
I20260619 16:47:59.774796 19488 heartbeater.cc:503] Master 127.0.97.62:36389 was elected leader, sending a full tablet report...
I20260619 16:47:59.774756 19636 raft_consensus.cc:348] T defd28bf1f3844329ed93a775d69e05b P 69975283742140389a2325c7b88e814b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69975283742140389a2325c7b88e814b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37667 } } peers { permanent_uuid: "bae81e16ae254fb9b3b69fc6fe0c515b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37867 } } peers { permanent_uuid: "214cb7fd381a4288bb71cd4e11baee5c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45361 } }
I20260619 16:47:59.775166 19636 raft_consensus.cc:374] T defd28bf1f3844329ed93a775d69e05b P 69975283742140389a2325c7b88e814b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:47:59.775250 19636 raft_consensus.cc:729] T defd28bf1f3844329ed93a775d69e05b P 69975283742140389a2325c7b88e814b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69975283742140389a2325c7b88e814b, State: Initialized, Role: FOLLOWER
I20260619 16:47:59.775822 19636 consensus_queue.cc:260] T defd28bf1f3844329ed93a775d69e05b P 69975283742140389a2325c7b88e814b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69975283742140389a2325c7b88e814b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37667 } } peers { permanent_uuid: "bae81e16ae254fb9b3b69fc6fe0c515b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37867 } } peers { permanent_uuid: "214cb7fd381a4288bb71cd4e11baee5c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45361 } }
I20260619 16:47:59.776859 19357 heartbeater.cc:503] Master 127.0.97.62:36389 was elected leader, sending a full tablet report...
I20260619 16:47:59.777454 19636 ts_tablet_manager.cc:1428] T defd28bf1f3844329ed93a775d69e05b P 69975283742140389a2325c7b88e814b: Time spent starting tablet: real 0.012s user 0.009s sys 0.005s
I20260619 16:47:59.780757 19640 raft_consensus.cc:348] T defd28bf1f3844329ed93a775d69e05b P 214cb7fd381a4288bb71cd4e11baee5c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69975283742140389a2325c7b88e814b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37667 } } peers { permanent_uuid: "bae81e16ae254fb9b3b69fc6fe0c515b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37867 } } peers { permanent_uuid: "214cb7fd381a4288bb71cd4e11baee5c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45361 } }
I20260619 16:47:59.781176 19640 raft_consensus.cc:374] T defd28bf1f3844329ed93a775d69e05b P 214cb7fd381a4288bb71cd4e11baee5c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:47:59.781278 19640 raft_consensus.cc:729] T defd28bf1f3844329ed93a775d69e05b P 214cb7fd381a4288bb71cd4e11baee5c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 214cb7fd381a4288bb71cd4e11baee5c, State: Initialized, Role: FOLLOWER
I20260619 16:47:59.781888 19640 consensus_queue.cc:260] T defd28bf1f3844329ed93a775d69e05b P 214cb7fd381a4288bb71cd4e11baee5c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69975283742140389a2325c7b88e814b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37667 } } peers { permanent_uuid: "bae81e16ae254fb9b3b69fc6fe0c515b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37867 } } peers { permanent_uuid: "214cb7fd381a4288bb71cd4e11baee5c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45361 } }
I20260619 16:47:59.782848 19618 heartbeater.cc:503] Master 127.0.97.62:36389 was elected leader, sending a full tablet report...
W20260619 16:47:59.783578 19358 tablet.cc:2369] T defd28bf1f3844329ed93a775d69e05b P 69975283742140389a2325c7b88e814b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 16:47:59.784291 19640 ts_tablet_manager.cc:1428] T defd28bf1f3844329ed93a775d69e05b P 214cb7fd381a4288bb71cd4e11baee5c: Time spent starting tablet: real 0.013s user 0.013s sys 0.000s
I20260619 16:47:59.845297 19643 raft_consensus.cc:482] T defd28bf1f3844329ed93a775d69e05b P 69975283742140389a2325c7b88e814b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 16:47:59.845563 19643 raft_consensus.cc:504] T defd28bf1f3844329ed93a775d69e05b P 69975283742140389a2325c7b88e814b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69975283742140389a2325c7b88e814b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37667 } } peers { permanent_uuid: "bae81e16ae254fb9b3b69fc6fe0c515b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37867 } } peers { permanent_uuid: "214cb7fd381a4288bb71cd4e11baee5c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45361 } }
I20260619 16:47:59.847318 19643 leader_election.cc:288] T defd28bf1f3844329ed93a775d69e05b P 69975283742140389a2325c7b88e814b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bae81e16ae254fb9b3b69fc6fe0c515b (127.0.97.2:37867), 214cb7fd381a4288bb71cd4e11baee5c (127.0.97.3:45361)
I20260619 16:47:59.854920 19633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:47:59.847751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:39959 (local address 127.0.97.2:37867)
0619 16:47:59.847992 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0619 16:47:59.847999 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:47:59.848410 (+ 411us) server_negotiation.cc:407] Connection header received
0619 16:47:59.848653 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:47:59.848660 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:47:59.848744 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:47:59.848841 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:47:59.849423 (+ 582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:47:59.850435 (+ 1012us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:47:59.852843 (+ 2408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:47:59.854100 (+ 1257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:47:59.854246 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:47:59.854367 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0619 16:47:59.854561 (+ 194us) server_negotiation.cc:299] Negotiation successful
0619 16:47:59.854671 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260619 16:47:59.855029 19645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:47:59.847685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:37867 (local address 127.0.97.1:39959)
0619 16:47:59.848309 (+ 624us) negotiation.cc:107] Waiting for socket to connect
0619 16:47:59.848328 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 16:47:59.848486 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:47:59.848879 (+ 393us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:47:59.848883 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:47:59.848917 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:47:59.849289 (+ 372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:47:59.849301 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:47:59.850561 (+ 1260us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:47:59.850564 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:47:59.852656 (+ 2092us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:47:59.852671 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:47:59.854271 (+ 1600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:47:59.854275 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:47:59.854391 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:47:59.854425 (+ 34us) client_negotiation.cc:769] Sending connection context
0619 16:47:59.854519 (+ 94us) client_negotiation.cc:240] Negotiation successful
0619 16:47:59.854663 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":520,"mutex_wait_us":1,"thread_start_us":364,"threads_started":1}
I20260619 16:47:59.855228 19634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:47:59.848572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:32959 (local address 127.0.97.3:45361)
0619 16:47:59.848747 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0619 16:47:59.848752 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:47:59.849114 (+ 362us) server_negotiation.cc:407] Connection header received
0619 16:47:59.849268 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:47:59.849272 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:47:59.849359 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:47:59.849449 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:47:59.849913 (+ 464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:47:59.850793 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:47:59.853024 (+ 2231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:47:59.854319 (+ 1295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:47:59.854443 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:47:59.854536 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0619 16:47:59.854846 (+ 310us) server_negotiation.cc:299] Negotiation successful
0619 16:47:59.854969 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260619 16:47:59.855458 19646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:47:59.848422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:45361 (local address 127.0.97.1:32959)
0619 16:47:59.849015 (+ 593us) negotiation.cc:107] Waiting for socket to connect
0619 16:47:59.849032 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 16:47:59.849155 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:47:59.849466 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:47:59.849469 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:47:59.849487 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:47:59.849769 (+ 282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:47:59.849779 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:47:59.850901 (+ 1122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:47:59.850905 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:47:59.852866 (+ 1961us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:47:59.852875 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:47:59.854506 (+ 1631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:47:59.854512 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:47:59.854699 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:47:59.854727 (+ 28us) client_negotiation.cc:769] Sending connection context
0619 16:47:59.854808 (+ 81us) client_negotiation.cc:240] Negotiation successful
0619 16:47:59.854910 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":468,"mutex_wait_us":24,"thread_start_us":282,"threads_started":1}
I20260619 16:47:59.856107 19443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "defd28bf1f3844329ed93a775d69e05b" candidate_uuid: "69975283742140389a2325c7b88e814b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bae81e16ae254fb9b3b69fc6fe0c515b" is_pre_election: true
I20260619 16:47:59.856433 19573 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "defd28bf1f3844329ed93a775d69e05b" candidate_uuid: "69975283742140389a2325c7b88e814b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "214cb7fd381a4288bb71cd4e11baee5c" is_pre_election: true
I20260619 16:47:59.856688 19443 raft_consensus.cc:2454] T defd28bf1f3844329ed93a775d69e05b P bae81e16ae254fb9b3b69fc6fe0c515b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69975283742140389a2325c7b88e814b in term 0.
I20260619 16:47:59.856932 19573 raft_consensus.cc:2454] T defd28bf1f3844329ed93a775d69e05b P 214cb7fd381a4288bb71cd4e11baee5c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69975283742140389a2325c7b88e814b in term 0.
I20260619 16:47:59.857396 19246 leader_election.cc:302] T defd28bf1f3844329ed93a775d69e05b P 69975283742140389a2325c7b88e814b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 69975283742140389a2325c7b88e814b, bae81e16ae254fb9b3b69fc6fe0c515b; no voters:
I20260619 16:47:59.857751 19643 raft_consensus.cc:2781] T defd28bf1f3844329ed93a775d69e05b P 69975283742140389a2325c7b88e814b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 16:47:59.857877 19643 raft_consensus.cc:482] T defd28bf1f3844329ed93a775d69e05b P 69975283742140389a2325c7b88e814b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 16:47:59.858000 19643 raft_consensus.cc:3037] T defd28bf1f3844329ed93a775d69e05b P 69975283742140389a2325c7b88e814b [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:47:59.860054 19643 raft_consensus.cc:504] T defd28bf1f3844329ed93a775d69e05b P 69975283742140389a2325c7b88e814b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69975283742140389a2325c7b88e814b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37667 } } peers { permanent_uuid: "bae81e16ae254fb9b3b69fc6fe0c515b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37867 } } peers { permanent_uuid: "214cb7fd381a4288bb71cd4e11baee5c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45361 } }
I20260619 16:47:59.860566 19643 leader_election.cc:288] T defd28bf1f3844329ed93a775d69e05b P 69975283742140389a2325c7b88e814b [CANDIDATE]: Term 1 election: Requested vote from peers bae81e16ae254fb9b3b69fc6fe0c515b (127.0.97.2:37867), 214cb7fd381a4288bb71cd4e11baee5c (127.0.97.3:45361)
I20260619 16:47:59.860960 19443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "defd28bf1f3844329ed93a775d69e05b" candidate_uuid: "69975283742140389a2325c7b88e814b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bae81e16ae254fb9b3b69fc6fe0c515b"
I20260619 16:47:59.861032 19573 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "defd28bf1f3844329ed93a775d69e05b" candidate_uuid: "69975283742140389a2325c7b88e814b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "214cb7fd381a4288bb71cd4e11baee5c"
I20260619 16:47:59.861114 19443 raft_consensus.cc:3037] T defd28bf1f3844329ed93a775d69e05b P bae81e16ae254fb9b3b69fc6fe0c515b [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:47:59.861218 19573 raft_consensus.cc:3037] T defd28bf1f3844329ed93a775d69e05b P 214cb7fd381a4288bb71cd4e11baee5c [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:47:59.863272 19443 raft_consensus.cc:2454] T defd28bf1f3844329ed93a775d69e05b P bae81e16ae254fb9b3b69fc6fe0c515b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 69975283742140389a2325c7b88e814b in term 1.
I20260619 16:47:59.863363 19573 raft_consensus.cc:2454] T defd28bf1f3844329ed93a775d69e05b P 214cb7fd381a4288bb71cd4e11baee5c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 69975283742140389a2325c7b88e814b in term 1.
I20260619 16:47:59.863654 19246 leader_election.cc:302] T defd28bf1f3844329ed93a775d69e05b P 69975283742140389a2325c7b88e814b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 69975283742140389a2325c7b88e814b, bae81e16ae254fb9b3b69fc6fe0c515b; no voters:
I20260619 16:47:59.863873 19643 raft_consensus.cc:2781] T defd28bf1f3844329ed93a775d69e05b P 69975283742140389a2325c7b88e814b [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:47:59.864333 19643 raft_consensus.cc:686] T defd28bf1f3844329ed93a775d69e05b P 69975283742140389a2325c7b88e814b [term 1 LEADER]: Becoming Leader. State: Replica: 69975283742140389a2325c7b88e814b, State: Running, Role: LEADER
I20260619 16:47:59.864861 19643 consensus_queue.cc:237] T defd28bf1f3844329ed93a775d69e05b P 69975283742140389a2325c7b88e814b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69975283742140389a2325c7b88e814b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37667 } } peers { permanent_uuid: "bae81e16ae254fb9b3b69fc6fe0c515b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37867 } } peers { permanent_uuid: "214cb7fd381a4288bb71cd4e11baee5c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45361 } }
I20260619 16:47:59.869134 19171 catalog_manager.cc:5495] T defd28bf1f3844329ed93a775d69e05b P 69975283742140389a2325c7b88e814b reported cstate change: term changed from 0 to 1, leader changed from <none> to 69975283742140389a2325c7b88e814b (127.0.97.1). New cstate: current_term: 1 leader_uuid: "69975283742140389a2325c7b88e814b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69975283742140389a2325c7b88e814b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37667 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bae81e16ae254fb9b3b69fc6fe0c515b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37867 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "214cb7fd381a4288bb71cd4e11baee5c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45361 } health_report { overall_health: UNKNOWN } } }
I20260619 16:47:59.889483 19632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:47:59.884081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51658 (local address 127.0.97.1:37667)
0619 16:47:59.884242 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0619 16:47:59.884247 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:47:59.884279 (+ 32us) server_negotiation.cc:407] Connection header received
0619 16:47:59.884354 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:47:59.884357 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:47:59.884472 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:47:59.884616 (+ 144us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 16:47:59.884984 (+ 368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:47:59.885670 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:47:59.887027 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:47:59.887442 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:47:59.887549 (+ 107us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:47:59.888097 (+ 548us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 16:47:59.888739 (+ 642us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 16:47:59.888904 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0619 16:47:59.889144 (+ 240us) server_negotiation.cc:299] Negotiation successful
0619 16:47:59.889190 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260619 16:47:59.898137 19573 raft_consensus.cc:1261] T defd28bf1f3844329ed93a775d69e05b P 214cb7fd381a4288bb71cd4e11baee5c [term 1 FOLLOWER]: Refusing update from remote peer 69975283742140389a2325c7b88e814b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:47:59.898172 19443 raft_consensus.cc:1261] T defd28bf1f3844329ed93a775d69e05b P bae81e16ae254fb9b3b69fc6fe0c515b [term 1 FOLLOWER]: Refusing update from remote peer 69975283742140389a2325c7b88e814b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:47:59.899029 19647 consensus_queue.cc:1035] T defd28bf1f3844329ed93a775d69e05b P 69975283742140389a2325c7b88e814b [LEADER]: Connected to new peer: Peer: permanent_uuid: "bae81e16ae254fb9b3b69fc6fe0c515b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37867 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:47:59.899407 19643 consensus_queue.cc:1035] T defd28bf1f3844329ed93a775d69e05b P 69975283742140389a2325c7b88e814b [LEADER]: Connected to new peer: Peer: permanent_uuid: "214cb7fd381a4288bb71cd4e11baee5c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:47:59.910526 19650 mvcc.cc:205] Tried to move back new op lower bound from 7298611936853372928 to 7298611936730390528. Current Snapshot: MvccSnapshot[applied={T|T < 7298611936853372928}]
W20260619 16:47:59.913128 19619 tablet.cc:2369] T defd28bf1f3844329ed93a775d69e05b P 214cb7fd381a4288bb71cd4e11baee5c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 16:47:59.928103 19170 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33598:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:47:59.928391 19170 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33598:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:47:59.932396 19170 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 90d117b5d0eb4a4687558819b254295c: Sending DeleteTablet for 3 replicas of tablet defd28bf1f3844329ed93a775d69e05b
I20260619 16:47:59.933579 19292 tablet_service.cc:1514] Processing DeleteTablet for tablet defd28bf1f3844329ed93a775d69e05b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:47:59 UTC) from {username='slave'} at 127.0.0.1:51654
I20260619 16:47:59.933794 19423 tablet_service.cc:1514] Processing DeleteTablet for tablet defd28bf1f3844329ed93a775d69e05b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:47:59 UTC) from {username='slave'} at 127.0.0.1:36356
I20260619 16:47:59.933833 19553 tablet_service.cc:1514] Processing DeleteTablet for tablet defd28bf1f3844329ed93a775d69e05b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:47:59 UTC) from {username='slave'} at 127.0.0.1:37706
I20260619 16:47:59.935593 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu pid 19230
I20260619 16:47:59.946161 19661 tablet_replica.cc:330] T defd28bf1f3844329ed93a775d69e05b P 69975283742140389a2325c7b88e814b: stopping tablet replica
I20260619 16:47:59.946664 19661 raft_consensus.cc:2229] T defd28bf1f3844329ed93a775d69e05b P 69975283742140389a2325c7b88e814b [term 1 LEADER]: Raft consensus shutting down.
I20260619 16:47:59.947467 19661 raft_consensus.cc:2258] T defd28bf1f3844329ed93a775d69e05b P 69975283742140389a2325c7b88e814b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:47:59.949941 19661 ts_tablet_manager.cc:1905] T defd28bf1f3844329ed93a775d69e05b P 69975283742140389a2325c7b88e814b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:47:59.954370 19663 tablet_replica.cc:330] T defd28bf1f3844329ed93a775d69e05b P 214cb7fd381a4288bb71cd4e11baee5c: stopping tablet replica
I20260619 16:47:59.954838 19663 raft_consensus.cc:2229] T defd28bf1f3844329ed93a775d69e05b P 214cb7fd381a4288bb71cd4e11baee5c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 16:47:59.955379 19663 raft_consensus.cc:2258] T defd28bf1f3844329ed93a775d69e05b P 214cb7fd381a4288bb71cd4e11baee5c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:47:59.956348 19632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:47:59.937585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51660 (local address 127.0.97.1:37667)
0619 16:47:59.937809 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0619 16:47:59.937816 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:47:59.942524 (+ 4708us) server_negotiation.cc:407] Connection header received
0619 16:47:59.943506 (+ 982us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:47:59.943513 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:47:59.943658 (+ 145us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:47:59.943743 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:47:59.944888 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:47:59.946121 (+ 1233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:47:59.949542 (+ 3421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:47:59.950158 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:47:59.950254 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:47:59.952562 (+ 2308us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:47:59.952573 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:47:59.952593 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:47:59.953381 (+ 788us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:47:59.953448 (+ 67us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:47:59.953472 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:47:59.953554 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0619 16:47:59.955827 (+ 2273us) server_negotiation.cc:299] Negotiation successful
0619 16:47:59.955961 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260619 16:47:59.957593 19661 ts_tablet_manager.cc:1918] T defd28bf1f3844329ed93a775d69e05b P 69975283742140389a2325c7b88e814b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 16:47:59.957628 19352 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:47:59.959652 19663 ts_tablet_manager.cc:1905] T defd28bf1f3844329ed93a775d69e05b P 214cb7fd381a4288bb71cd4e11baee5c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:47:59.960106 19661 log.cc:1196] T defd28bf1f3844329ed93a775d69e05b P 69975283742140389a2325c7b88e814b: Deleting WAL directory at /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781887385903558-388-0/minicluster-data/ts-0/wal/wals/defd28bf1f3844329ed93a775d69e05b
I20260619 16:47:59.960676 19662 tablet_replica.cc:330] T defd28bf1f3844329ed93a775d69e05b P bae81e16ae254fb9b3b69fc6fe0c515b: stopping tablet replica
I20260619 16:47:59.961007 19662 raft_consensus.cc:2229] T defd28bf1f3844329ed93a775d69e05b P bae81e16ae254fb9b3b69fc6fe0c515b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 16:47:59.961400 19662 raft_consensus.cc:2258] T defd28bf1f3844329ed93a775d69e05b P bae81e16ae254fb9b3b69fc6fe0c515b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:47:59.963088 19662 ts_tablet_manager.cc:1905] T defd28bf1f3844329ed93a775d69e05b P bae81e16ae254fb9b3b69fc6fe0c515b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:47:59.966159 19663 ts_tablet_manager.cc:1918] T defd28bf1f3844329ed93a775d69e05b P 214cb7fd381a4288bb71cd4e11baee5c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 16:47:59.966316 19663 log.cc:1196] T defd28bf1f3844329ed93a775d69e05b P 214cb7fd381a4288bb71cd4e11baee5c: Deleting WAL directory at /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781887385903558-388-0/minicluster-data/ts-2/wal/wals/defd28bf1f3844329ed93a775d69e05b
I20260619 16:47:59.966812 19663 ts_tablet_manager.cc:1939] T defd28bf1f3844329ed93a775d69e05b P 214cb7fd381a4288bb71cd4e11baee5c: Deleting consensus metadata
I20260619 16:47:59.967752 19662 ts_tablet_manager.cc:1918] T defd28bf1f3844329ed93a775d69e05b P bae81e16ae254fb9b3b69fc6fe0c515b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 16:47:59.967881 19662 log.cc:1196] T defd28bf1f3844329ed93a775d69e05b P bae81e16ae254fb9b3b69fc6fe0c515b: Deleting WAL directory at /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781887385903558-388-0/minicluster-data/ts-1/wal/wals/defd28bf1f3844329ed93a775d69e05b
I20260619 16:47:59.968262 19662 ts_tablet_manager.cc:1939] T defd28bf1f3844329ed93a775d69e05b P bae81e16ae254fb9b3b69fc6fe0c515b: Deleting consensus metadata
I20260619 16:47:59.968628 19156 catalog_manager.cc:4841] TS 214cb7fd381a4288bb71cd4e11baee5c (127.0.97.3:45361): tablet defd28bf1f3844329ed93a775d69e05b (table test-table [id=7eda522923db41159d1432f4985b5a4d]) successfully deleted
I20260619 16:47:59.969573 19156 catalog_manager.cc:4841] TS bae81e16ae254fb9b3b69fc6fe0c515b (127.0.97.2:37867): tablet defd28bf1f3844329ed93a775d69e05b (table test-table [id=7eda522923db41159d1432f4985b5a4d]) successfully deleted
I20260619 16:48:00.055181 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu with pid 19230
I20260619 16:48:00.055254 19661 ts_tablet_manager.cc:1939] T defd28bf1f3844329ed93a775d69e05b P 69975283742140389a2325c7b88e814b: Deleting consensus metadata
I20260619 16:48:00.069427 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu pid 19361
W20260619 16:48:00.069593 19157 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.97.1:37667 (error 108)
I20260619 16:48:00.071477 19635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:48:00.070971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:37667 (local address 127.0.0.1:51676)
0619 16:48:00.071159 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0619 16:48:00.071287 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:37667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260619 16:48:00.071702 19157 catalog_manager.cc:4584] TS 69975283742140389a2325c7b88e814b (127.0.97.1:37667): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet defd28bf1f3844329ed93a775d69e05b: Network error: Client connection negotiation failed: client connection to 127.0.97.1:37667: connect: Connection refused (error 111)
I20260619 16:48:00.076378 19633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:48:00.070486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36368 (local address 127.0.97.2:37867)
0619 16:48:00.070674 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0619 16:48:00.070678 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:48:00.070692 (+ 14us) server_negotiation.cc:407] Connection header received
0619 16:48:00.070743 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:48:00.070748 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:48:00.070875 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:48:00.070977 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:48:00.071435 (+ 458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:48:00.072316 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:48:00.073531 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:48:00.074054 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:48:00.074147 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:48:00.075065 (+ 918us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:48:00.075074 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:48:00.075087 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:48:00.075644 (+ 557us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:48:00.075688 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:48:00.075707 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:48:00.075801 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0619 16:48:00.075988 (+ 187us) server_negotiation.cc:299] Negotiation successful
0619 16:48:00.076083 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260619 16:48:00.076983 19483 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:48:00.125635 19635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:48:00.125106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:37667 (local address 127.0.0.1:51690)
0619 16:48:00.125354 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0619 16:48:00.125461 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:37667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106}
I20260619 16:48:00.126693 19635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:48:00.126336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:37667 (local address 127.0.0.1:51698)
0619 16:48:00.126493 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0619 16:48:00.126571 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:37667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260619 16:48:00.164501 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu with pid 19361
I20260619 16:48:00.177412 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu pid 19491
I20260619 16:48:00.183763 19634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:48:00.178219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37714 (local address 127.0.97.3:45361)
0619 16:48:00.178397 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0619 16:48:00.178429 (+ 32us) server_negotiation.cc:399] Waiting for connection header
0619 16:48:00.178448 (+ 19us) server_negotiation.cc:407] Connection header received
0619 16:48:00.178504 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:48:00.178507 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:48:00.178596 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:48:00.178683 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:48:00.179026 (+ 343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:48:00.179765 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:48:00.181068 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:48:00.181446 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:48:00.181536 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:48:00.182570 (+ 1034us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:48:00.182579 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:48:00.182593 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:48:00.183126 (+ 533us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:48:00.183171 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:48:00.183190 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:48:00.183273 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0619 16:48:00.183439 (+ 166us) server_negotiation.cc:299] Negotiation successful
0619 16:48:00.183529 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260619 16:48:00.184386 19613 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:48:00.187001 19635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:48:00.186583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:37667 (local address 127.0.0.1:51714)
0619 16:48:00.186769 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0619 16:48:00.186857 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:37667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260619 16:48:00.188328 19635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:48:00.187879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:37667 (local address 127.0.0.1:51728)
0619 16:48:00.188056 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0619 16:48:00.188170 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:37667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260619 16:48:00.269148 19635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:48:00.268620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:37667 (local address 127.0.0.1:51742)
0619 16:48:00.268861 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0619 16:48:00.268972 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:37667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260619 16:48:00.270372 19635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:48:00.269904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:37667 (local address 127.0.0.1:51750)
0619 16:48:00.270129 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0619 16:48:00.270222 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:37667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109}
I20260619 16:48:00.273643 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu with pid 19491
I20260619 16:48:00.286513 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu pid 19140
I20260619 16:48:00.287505 19201 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:48:00.394748 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu with pid 19140
2026-06-19T16:48:00Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (2618 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887385903558-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 16:48:00 dist-test-slave-wvz7 krb5kdc[19672](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 16:48:00 dist-test-slave-wvz7 krb5kdc[19672](info): set up 2 sockets
Jun 19 16:48:00 dist-test-slave-wvz7 krb5kdc[19672](info): commencing operation
krb5kdc: starting...
W20260619 16:48:01.417666 388 mini_kdc.cc:121] Time spent starting KDC: real 0.987s user 0.003s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887385903558-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887385903558-388-0/krb5kdc/test-user.keytab.
Jun 19 16:48:01 dist-test-slave-wvz7 krb5kdc[19672](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887681, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T16:48:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T16:48:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887385903558-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887385903558-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260619 16:48:01.657378 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu
/tmp/dist-test-taskEs2EH8/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887385903558-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887385903558-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887385903558-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887385903558-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34359
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887385903558-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40055
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:34359
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887385903558-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887385903558-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887385903558-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:48:01.973940 19685 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:48:01.974233 19685 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:48:01.974367 19685 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:48:01.981874 19685 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:48:01.981981 19685 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:48:01.982055 19685 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:48:01.982131 19685 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:48:01.982188 19685 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:48:01.992025 19685 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40055
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887385903558-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887385903558-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:34359
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887385903558-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34359
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887385903558-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887385903558-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:48:01.993347 19685 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:48:01.994658 19685 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:48:02.006407 19690 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:48:02.006343 19693 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:48:02.006412 19691 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:48:02.007017 19685 server_base.cc:1034] running on GCE node
I20260619 16:48:02.007789 19685 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:48:02.009255 19685 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:48:02.010449 19685 hybrid_clock.cc:648] HybridClock initialized: now 1781887682010392 us; error 41 us; skew 500 ppm
I20260619 16:48:02.010767 19685 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:48:02 dist-test-slave-wvz7 krb5kdc[19672](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887682, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:48:02.013636 19685 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260619 16:48:02.015089 19685 webserver.cc:460] Webserver started at http://127.0.97.62:44117/ using document root <none> and password file <none>
I20260619 16:48:02.015749 19685 fs_manager.cc:359] Metadata directory not provided
I20260619 16:48:02.015861 19685 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:48:02.016158 19685 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:48:02.019224 19685 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887385903558-388-0/minicluster-data/master-0/data/instance:
uuid: "a8f4fe346a4c4a03a94b2e779e3daef0"
format_stamp: "Formatted at 2026-06-19 16:48:02 on dist-test-slave-wvz7"
server_key: "53eb4c1bc9392ce0e84e5a14cb143f52"
server_key_iv: "5f08245258829fce8e06065e52a1e271"
server_key_version: "encryptionkey@0"
I20260619 16:48:02.019950 19685 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887385903558-388-0/minicluster-data/master-0/wal/instance:
uuid: "a8f4fe346a4c4a03a94b2e779e3daef0"
format_stamp: "Formatted at 2026-06-19 16:48:02 on dist-test-slave-wvz7"
server_key: "53eb4c1bc9392ce0e84e5a14cb143f52"
server_key_iv: "5f08245258829fce8e06065e52a1e271"
server_key_version: "encryptionkey@0"
I20260619 16:48:02.025018 19685 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260619 16:48:02.028436 19700 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:48:02.029906 19685 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 16:48:02.030061 19685 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887385903558-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887385903558-388-0/minicluster-data/master-0/wal
uuid: "a8f4fe346a4c4a03a94b2e779e3daef0"
format_stamp: "Formatted at 2026-06-19 16:48:02 on dist-test-slave-wvz7"
server_key: "53eb4c1bc9392ce0e84e5a14cb143f52"
server_key_iv: "5f08245258829fce8e06065e52a1e271"
server_key_version: "encryptionkey@0"
I20260619 16:48:02.030244 19685 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887385903558-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887385903558-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887385903558-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:48:02.051678 19685 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:48:02.055744 19685 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:48:02.056015 19685 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:48:02.073338 19685 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:34359
I20260619 16:48:02.073369 19751 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:34359 every 8 connection(s)
I20260619 16:48:02.074714 19685 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887385903558-388-0/minicluster-data/master-0/data/info.pb
I20260619 16:48:02.078502 19752 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:48:02.080015 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu as pid 19685
I20260619 16:48:02.080171 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887385903558-388-0/minicluster-data/master-0/wal/instance
I20260619 16:48:02.080621 388 external_mini_cluster.cc:1428] Setting key 79c16631e31306cac264703ee13e1578
I20260619 16:48:02.092425 19752 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a8f4fe346a4c4a03a94b2e779e3daef0: Bootstrap starting.
Jun 19 16:48:02 dist-test-slave-wvz7 krb5kdc[19672](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887681, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 16:48:02.095657 19752 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a8f4fe346a4c4a03a94b2e779e3daef0: Neither blocks nor log segments found. Creating new log.
I20260619 16:48:02.096947 19752 log.cc:824] T 00000000000000000000000000000000 P a8f4fe346a4c4a03a94b2e779e3daef0: Log is configured to *not* fsync() on all Append() calls
I20260619 16:48:02.100378 19752 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a8f4fe346a4c4a03a94b2e779e3daef0: No bootstrap required, opened a new log
I20260619 16:48:02.104874 19755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:48:02.082480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38774 (local address 127.0.97.62:34359)
0619 16:48:02.083286 (+ 806us) server_negotiation.cc:206] Beginning negotiation
0619 16:48:02.083301 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 16:48:02.083352 (+ 51us) server_negotiation.cc:407] Connection header received
0619 16:48:02.085021 (+ 1669us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:48:02.085057 (+ 36us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:48:02.085620 (+ 563us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:48:02.086115 (+ 495us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:48:02.087379 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:48:02.089359 (+ 1980us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:48:02.090733 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:48:02.091750 (+ 1017us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:48:02.091930 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:48:02.096095 (+ 4165us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:48:02.096107 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:48:02.096126 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:48:02.096173 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:48:02.099944 (+ 3771us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:48:02.100725 (+ 781us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:48:02.100734 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:48:02.100743 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:48:02.100992 (+ 249us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:48:02.101543 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:48:02.101548 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:48:02.101551 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:48:02.102282 (+ 731us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:48:02.102608 (+ 326us) server_negotiation.cc:1035] Waiting for connection context
0619 16:48:02.102950 (+ 342us) server_negotiation.cc:299] Negotiation successful
0619 16:48:02.103404 (+ 454us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":563,"thread_start_us":335,"threads_started":1}
I20260619 16:48:02.109966 19752 raft_consensus.cc:348] T 00000000000000000000000000000000 P a8f4fe346a4c4a03a94b2e779e3daef0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8f4fe346a4c4a03a94b2e779e3daef0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34359 } }
I20260619 16:48:02.110312 19752 raft_consensus.cc:374] T 00000000000000000000000000000000 P a8f4fe346a4c4a03a94b2e779e3daef0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:48:02.110391 19752 raft_consensus.cc:729] T 00000000000000000000000000000000 P a8f4fe346a4c4a03a94b2e779e3daef0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a8f4fe346a4c4a03a94b2e779e3daef0, State: Initialized, Role: FOLLOWER
I20260619 16:48:02.110877 19752 consensus_queue.cc:260] T 00000000000000000000000000000000 P a8f4fe346a4c4a03a94b2e779e3daef0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8f4fe346a4c4a03a94b2e779e3daef0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34359 } }
I20260619 16:48:02.111081 19752 raft_consensus.cc:388] T 00000000000000000000000000000000 P a8f4fe346a4c4a03a94b2e779e3daef0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:48:02.111177 19752 raft_consensus.cc:482] T 00000000000000000000000000000000 P a8f4fe346a4c4a03a94b2e779e3daef0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:48:02.111333 19752 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a8f4fe346a4c4a03a94b2e779e3daef0 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:48:02.113703 19752 raft_consensus.cc:504] T 00000000000000000000000000000000 P a8f4fe346a4c4a03a94b2e779e3daef0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8f4fe346a4c4a03a94b2e779e3daef0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34359 } }
I20260619 16:48:02.114257 19752 leader_election.cc:302] T 00000000000000000000000000000000 P a8f4fe346a4c4a03a94b2e779e3daef0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a8f4fe346a4c4a03a94b2e779e3daef0; no voters:
I20260619 16:48:02.114872 19752 leader_election.cc:288] T 00000000000000000000000000000000 P a8f4fe346a4c4a03a94b2e779e3daef0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 16:48:02.115011 19757 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a8f4fe346a4c4a03a94b2e779e3daef0 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:48:02.115639 19757 raft_consensus.cc:686] T 00000000000000000000000000000000 P a8f4fe346a4c4a03a94b2e779e3daef0 [term 1 LEADER]: Becoming Leader. State: Replica: a8f4fe346a4c4a03a94b2e779e3daef0, State: Running, Role: LEADER
I20260619 16:48:02.116168 19757 consensus_queue.cc:237] T 00000000000000000000000000000000 P a8f4fe346a4c4a03a94b2e779e3daef0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8f4fe346a4c4a03a94b2e779e3daef0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34359 } }
I20260619 16:48:02.116809 19752 sys_catalog.cc:564] T 00000000000000000000000000000000 P a8f4fe346a4c4a03a94b2e779e3daef0 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:48:02.118881 19759 sys_catalog.cc:455] T 00000000000000000000000000000000 P a8f4fe346a4c4a03a94b2e779e3daef0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a8f4fe346a4c4a03a94b2e779e3daef0. Latest consensus state: current_term: 1 leader_uuid: "a8f4fe346a4c4a03a94b2e779e3daef0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8f4fe346a4c4a03a94b2e779e3daef0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34359 } } }
I20260619 16:48:02.119252 19759 sys_catalog.cc:458] T 00000000000000000000000000000000 P a8f4fe346a4c4a03a94b2e779e3daef0 [sys.catalog]: This master's current role is: LEADER
I20260619 16:48:02.119999 19758 sys_catalog.cc:455] T 00000000000000000000000000000000 P a8f4fe346a4c4a03a94b2e779e3daef0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a8f4fe346a4c4a03a94b2e779e3daef0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8f4fe346a4c4a03a94b2e779e3daef0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34359 } } }
I20260619 16:48:02.120373 19758 sys_catalog.cc:458] T 00000000000000000000000000000000 P a8f4fe346a4c4a03a94b2e779e3daef0 [sys.catalog]: This master's current role is: LEADER
I20260619 16:48:02.124415 19764 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:48:02.129494 19764 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:48:02.137260 19764 catalog_manager.cc:1347] Generated new cluster ID: d34ce31ba88a4e3e9344dfaebb753010
I20260619 16:48:02.137373 19764 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:48:02.150646 19764 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 16:48:02.152522 19764 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:48:02.184239 19764 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a8f4fe346a4c4a03a94b2e779e3daef0: Generated new TSK 0
I20260619 16:48:02.185380 19764 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887385903558-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887385903558-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260619 16:48:02.244498 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu
/tmp/dist-test-taskEs2EH8/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887385903558-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887385903558-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887385903558-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887385903558-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34359
--keytab_file=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887385903558-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40055
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887385903558-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887385903558-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887385903558-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:48:02.561306 19778 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:48:02.561580 19778 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:48:02.561703 19778 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:48:02.568879 19778 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:48:02.568991 19778 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:48:02.569110 19778 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260619 16:48:02.578845 19778 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40055
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887385903558-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887385903558-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887385903558-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887385903558-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34359
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887385903558-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:48:02.580036 19778 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:48:02.581233 19778 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:48:02.593528 19783 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:48:02.593444 19784 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:48:02.593485 19786 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:48:02.594345 19778 server_base.cc:1034] running on GCE node
I20260619 16:48:02.594924 19778 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:48:02.595826 19778 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:48:02.597026 19778 hybrid_clock.cc:648] HybridClock initialized: now 1781887682596982 us; error 37 us; skew 500 ppm
I20260619 16:48:02.597332 19778 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:48:02 dist-test-slave-wvz7 krb5kdc[19672](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887682, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:48:02.600332 19778 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260619 16:48:02.601651 19778 webserver.cc:460] Webserver started at http://127.0.97.1:41239/ using document root <none> and password file <none>
I20260619 16:48:02.602311 19778 fs_manager.cc:359] Metadata directory not provided
I20260619 16:48:02.602421 19778 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:48:02.602691 19778 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:48:02.605468 19778 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887385903558-388-0/minicluster-data/ts-0/data/instance:
uuid: "95f90b972b354f9a9c09b2b2432b0127"
format_stamp: "Formatted at 2026-06-19 16:48:02 on dist-test-slave-wvz7"
server_key: "ddafe40841313ee460658d38e6d7d6a1"
server_key_iv: "17c00851378ac91c21c7d9d7c001f1b8"
server_key_version: "encryptionkey@0"
I20260619 16:48:02.606267 19778 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887385903558-388-0/minicluster-data/ts-0/wal/instance:
uuid: "95f90b972b354f9a9c09b2b2432b0127"
format_stamp: "Formatted at 2026-06-19 16:48:02 on dist-test-slave-wvz7"
server_key: "ddafe40841313ee460658d38e6d7d6a1"
server_key_iv: "17c00851378ac91c21c7d9d7c001f1b8"
server_key_version: "encryptionkey@0"
I20260619 16:48:02.611047 19778 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260619 16:48:02.614405 19793 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:48:02.615770 19778 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 16:48:02.615918 19778 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887385903558-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887385903558-388-0/minicluster-data/ts-0/wal
uuid: "95f90b972b354f9a9c09b2b2432b0127"
format_stamp: "Formatted at 2026-06-19 16:48:02 on dist-test-slave-wvz7"
server_key: "ddafe40841313ee460658d38e6d7d6a1"
server_key_iv: "17c00851378ac91c21c7d9d7c001f1b8"
server_key_version: "encryptionkey@0"
I20260619 16:48:02.616098 19778 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887385903558-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887385903558-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887385903558-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:48:02.628659 19778 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:48:02.632783 19778 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:48:02.633045 19778 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:48:02.634125 19778 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:48:02.635560 19778 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:48:02.635635 19778 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:48:02.635746 19778 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:48:02.635802 19778 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:48:02.665659 19778 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:34093
I20260619 16:48:02.665699 19905 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:34093 every 8 connection(s)
I20260619 16:48:02.666904 19778 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887385903558-388-0/minicluster-data/ts-0/data/info.pb
I20260619 16:48:02.669839 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu as pid 19778
I20260619 16:48:02.670018 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887385903558-388-0/minicluster-data/ts-0/wal/instance
I20260619 16:48:02.670450 388 external_mini_cluster.cc:1428] Setting key f785ce226b1b14ce4a4fa712ccfdfc8b
Jun 19 16:48:02 dist-test-slave-wvz7 krb5kdc[19672](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887682, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 16:48:02.688666 19908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:48:02.669564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:58313 (local address 127.0.97.62:34359)
0619 16:48:02.670068 (+ 504us) server_negotiation.cc:206] Beginning negotiation
0619 16:48:02.670074 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:48:02.671112 (+ 1038us) server_negotiation.cc:407] Connection header received
0619 16:48:02.673291 (+ 2179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:48:02.673295 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:48:02.673363 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:48:02.673455 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:48:02.675463 (+ 2008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:48:02.676215 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:48:02.678250 (+ 2035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:48:02.678681 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:48:02.678781 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:48:02.682796 (+ 4015us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:48:02.682802 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:48:02.682805 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:48:02.682839 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:48:02.685408 (+ 2569us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:48:02.686322 (+ 914us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:48:02.686327 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:48:02.686329 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:48:02.686461 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:48:02.686926 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:48:02.686931 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:48:02.686942 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:48:02.687280 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:48:02.687396 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0619 16:48:02.688230 (+ 834us) server_negotiation.cc:299] Negotiation successful
0619 16:48:02.688407 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":377,"thread_start_us":262,"threads_started":1}
I20260619 16:48:02.690016 19909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:48:02.670111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34359 (local address 127.0.97.1:58313)
0619 16:48:02.670930 (+ 819us) negotiation.cc:107] Waiting for socket to connect
0619 16:48:02.670983 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0619 16:48:02.672891 (+ 1908us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:48:02.673755 (+ 864us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:48:02.673768 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:48:02.674499 (+ 731us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:48:02.675269 (+ 770us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:48:02.675292 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:48:02.676358 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:48:02.676362 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:48:02.678107 (+ 1745us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:48:02.678121 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:48:02.678894 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:48:02.678900 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:48:02.679150 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:48:02.679815 (+ 665us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:48:02.679843 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:48:02.682550 (+ 2707us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:48:02.685551 (+ 3001us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:48:02.685559 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:48:02.685578 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:48:02.686181 (+ 603us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:48:02.686560 (+ 379us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:48:02.686565 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:48:02.686569 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:48:02.686792 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:48:02.687398 (+ 606us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:48:02.687405 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:48:02.687814 (+ 409us) client_negotiation.cc:769] Sending connection context
0619 16:48:02.688123 (+ 309us) client_negotiation.cc:240] Negotiation successful
0619 16:48:02.688656 (+ 533us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":526,"thread_start_us":305,"threads_started":1}
I20260619 16:48:02.691524 19906 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34359
I20260619 16:48:02.691874 19906 heartbeater.cc:464] Registering TS with master...
I20260619 16:48:02.692742 19906 heartbeater.cc:511] Master 127.0.97.62:34359 requested a full tablet report, sending...
I20260619 16:48:02.695019 19717 ts_manager.cc:194] Registered new tserver with Master: 95f90b972b354f9a9c09b2b2432b0127 (127.0.97.1:34093)
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260619 16:48:02.696466 19717 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:58313
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887385903558-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887385903558-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260619 16:48:02.724466 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu
/tmp/dist-test-taskEs2EH8/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887385903558-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887385903558-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887385903558-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887385903558-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34359
--keytab_file=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887385903558-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40055
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887385903558-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887385903558-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887385903558-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:48:03.032403 19912 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:48:03.032680 19912 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:48:03.032819 19912 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:48:03.040038 19912 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:48:03.040170 19912 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:48:03.040328 19912 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260619 16:48:03.049845 19912 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40055
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887385903558-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887385903558-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887385903558-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887385903558-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34359
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887385903558-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:48:03.051075 19912 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:48:03.052371 19912 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:48:03.064563 19917 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:48:03.064630 19918 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:48:03.064599 19920 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:48:03.065562 19912 server_base.cc:1034] running on GCE node
I20260619 16:48:03.066198 19912 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:48:03.067086 19912 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:48:03.068312 19912 hybrid_clock.cc:648] HybridClock initialized: now 1781887683068264 us; error 43 us; skew 500 ppm
I20260619 16:48:03.068602 19912 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:48:03 dist-test-slave-wvz7 krb5kdc[19672](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887683, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:48:03.071534 19912 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260619 16:48:03.073009 19912 webserver.cc:460] Webserver started at http://127.0.97.2:44073/ using document root <none> and password file <none>
I20260619 16:48:03.073643 19912 fs_manager.cc:359] Metadata directory not provided
I20260619 16:48:03.073771 19912 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:48:03.074045 19912 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:48:03.076818 19912 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887385903558-388-0/minicluster-data/ts-1/data/instance:
uuid: "7acfaf949855430b82f50146a5fba04f"
format_stamp: "Formatted at 2026-06-19 16:48:03 on dist-test-slave-wvz7"
server_key: "a7443366ca93385b2c88a4887e6a6d7d"
server_key_iv: "545b795eab6b6b50f19fce4eece0a0fa"
server_key_version: "encryptionkey@0"
I20260619 16:48:03.077550 19912 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887385903558-388-0/minicluster-data/ts-1/wal/instance:
uuid: "7acfaf949855430b82f50146a5fba04f"
format_stamp: "Formatted at 2026-06-19 16:48:03 on dist-test-slave-wvz7"
server_key: "a7443366ca93385b2c88a4887e6a6d7d"
server_key_iv: "545b795eab6b6b50f19fce4eece0a0fa"
server_key_version: "encryptionkey@0"
I20260619 16:48:03.082454 19912 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.001s
I20260619 16:48:03.085835 19927 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:48:03.087193 19912 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260619 16:48:03.087342 19912 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887385903558-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887385903558-388-0/minicluster-data/ts-1/wal
uuid: "7acfaf949855430b82f50146a5fba04f"
format_stamp: "Formatted at 2026-06-19 16:48:03 on dist-test-slave-wvz7"
server_key: "a7443366ca93385b2c88a4887e6a6d7d"
server_key_iv: "545b795eab6b6b50f19fce4eece0a0fa"
server_key_version: "encryptionkey@0"
I20260619 16:48:03.087512 19912 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887385903558-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887385903558-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887385903558-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:48:03.123195 19912 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:48:03.127420 19912 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:48:03.127651 19912 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:48:03.128553 19912 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:48:03.129856 19912 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:48:03.129935 19912 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:48:03.130025 19912 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:48:03.130065 19912 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:48:03.159209 19912 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:44715
I20260619 16:48:03.159276 20039 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:44715 every 8 connection(s)
I20260619 16:48:03.160393 19912 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887385903558-388-0/minicluster-data/ts-1/data/info.pb
I20260619 16:48:03.166342 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu as pid 19912
I20260619 16:48:03.166520 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887385903558-388-0/minicluster-data/ts-1/wal/instance
I20260619 16:48:03.166960 388 external_mini_cluster.cc:1428] Setting key 8d6e194ce0b9127106a28ea254404757
Jun 19 16:48:03 dist-test-slave-wvz7 krb5kdc[19672](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887683, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 16:48:03.178807 19908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:48:03.162764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:56073 (local address 127.0.97.62:34359)
0619 16:48:03.162947 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0619 16:48:03.162952 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:48:03.164016 (+ 1064us) server_negotiation.cc:407] Connection header received
0619 16:48:03.165798 (+ 1782us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:48:03.165804 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:48:03.165890 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:48:03.165973 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:48:03.167471 (+ 1498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:48:03.168327 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:48:03.169851 (+ 1524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:48:03.170223 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:48:03.170328 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:48:03.173420 (+ 3092us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:48:03.173425 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:48:03.173428 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:48:03.173459 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:48:03.175938 (+ 2479us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:48:03.176612 (+ 674us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:48:03.176616 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:48:03.176619 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:48:03.176749 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:48:03.177134 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:48:03.177137 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:48:03.177139 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:48:03.177403 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:48:03.177535 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0619 16:48:03.178248 (+ 713us) server_negotiation.cc:299] Negotiation successful
0619 16:48:03.178482 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260619 16:48:03.179816 20042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:48:03.163226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34359 (local address 127.0.97.2:56073)
0619 16:48:03.163863 (+ 637us) negotiation.cc:107] Waiting for socket to connect
0619 16:48:03.163906 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0619 16:48:03.165534 (+ 1628us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:48:03.166211 (+ 677us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:48:03.166219 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:48:03.166752 (+ 533us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:48:03.167304 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:48:03.167321 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:48:03.168458 (+ 1137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:48:03.168461 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:48:03.169698 (+ 1237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:48:03.169711 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:48:03.170426 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:48:03.170434 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:48:03.170634 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:48:03.171197 (+ 563us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:48:03.171218 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:48:03.173225 (+ 2007us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:48:03.176065 (+ 2840us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:48:03.176073 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:48:03.176090 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:48:03.176504 (+ 414us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:48:03.176847 (+ 343us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:48:03.176852 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:48:03.176856 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:48:03.177034 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:48:03.177519 (+ 485us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:48:03.177526 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:48:03.177891 (+ 365us) client_negotiation.cc:769] Sending connection context
0619 16:48:03.178133 (+ 242us) client_negotiation.cc:240] Negotiation successful
0619 16:48:03.178676 (+ 543us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":421,"thread_start_us":260,"threads_started":1}
I20260619 16:48:03.181277 20040 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34359
I20260619 16:48:03.181648 20040 heartbeater.cc:464] Registering TS with master...
I20260619 16:48:03.182596 20040 heartbeater.cc:511] Master 127.0.97.62:34359 requested a full tablet report, sending...
I20260619 16:48:03.184244 19717 ts_manager.cc:194] Registered new tserver with Master: 7acfaf949855430b82f50146a5fba04f (127.0.97.2:44715)
I20260619 16:48:03.185408 19717 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:56073
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887385903558-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887385903558-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260619 16:48:03.223016 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu
/tmp/dist-test-taskEs2EH8/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887385903558-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887385903558-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887385903558-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887385903558-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34359
--keytab_file=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887385903558-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40055
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887385903558-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887385903558-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887385903558-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:48:03.534700 20045 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:48:03.534992 20045 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:48:03.535116 20045 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:48:03.542243 20045 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:48:03.542348 20045 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:48:03.542495 20045 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260619 16:48:03.552016 20045 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40055
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887385903558-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887385903558-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887385903558-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887385903558-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34359
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887385903558-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:48:03.553297 20045 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:48:03.554600 20045 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:48:03.567437 20050 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:48:03.567328 20051 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:48:03.567377 20053 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:48:03.567955 20045 server_base.cc:1034] running on GCE node
I20260619 16:48:03.568606 20045 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:48:03.569518 20045 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:48:03.570713 20045 hybrid_clock.cc:648] HybridClock initialized: now 1781887683570645 us; error 60 us; skew 500 ppm
I20260619 16:48:03.570994 20045 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:48:03 dist-test-slave-wvz7 krb5kdc[19672](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887683, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:48:03.573997 20045 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260619 16:48:03.575498 20045 webserver.cc:460] Webserver started at http://127.0.97.3:43455/ using document root <none> and password file <none>
I20260619 16:48:03.576162 20045 fs_manager.cc:359] Metadata directory not provided
I20260619 16:48:03.576279 20045 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:48:03.576609 20045 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:48:03.579591 20045 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887385903558-388-0/minicluster-data/ts-2/data/instance:
uuid: "0691559030a44924b787b5ed0bfe5dd3"
format_stamp: "Formatted at 2026-06-19 16:48:03 on dist-test-slave-wvz7"
server_key: "afe52e7a5ae00a8f2d47673136be4b00"
server_key_iv: "7a85a1be3d1016e5ce7a9eb2db03e486"
server_key_version: "encryptionkey@0"
I20260619 16:48:03.580356 20045 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887385903558-388-0/minicluster-data/ts-2/wal/instance:
uuid: "0691559030a44924b787b5ed0bfe5dd3"
format_stamp: "Formatted at 2026-06-19 16:48:03 on dist-test-slave-wvz7"
server_key: "afe52e7a5ae00a8f2d47673136be4b00"
server_key_iv: "7a85a1be3d1016e5ce7a9eb2db03e486"
server_key_version: "encryptionkey@0"
I20260619 16:48:03.585354 20045 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260619 16:48:03.588900 20060 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:48:03.590435 20045 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260619 16:48:03.590590 20045 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887385903558-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887385903558-388-0/minicluster-data/ts-2/wal
uuid: "0691559030a44924b787b5ed0bfe5dd3"
format_stamp: "Formatted at 2026-06-19 16:48:03 on dist-test-slave-wvz7"
server_key: "afe52e7a5ae00a8f2d47673136be4b00"
server_key_iv: "7a85a1be3d1016e5ce7a9eb2db03e486"
server_key_version: "encryptionkey@0"
I20260619 16:48:03.590775 20045 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887385903558-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887385903558-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887385903558-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:48:03.602871 20045 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:48:03.606920 20045 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:48:03.607189 20045 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:48:03.608048 20045 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:48:03.609285 20045 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:48:03.609365 20045 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:48:03.609460 20045 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:48:03.609506 20045 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:48:03.638863 20045 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:35767
I20260619 16:48:03.638957 20172 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:35767 every 8 connection(s)
I20260619 16:48:03.640084 20045 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887385903558-388-0/minicluster-data/ts-2/data/info.pb
I20260619 16:48:03.647925 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu as pid 20045
I20260619 16:48:03.648088 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887385903558-388-0/minicluster-data/ts-2/wal/instance
I20260619 16:48:03.648474 388 external_mini_cluster.cc:1428] Setting key 85cf045070ca20a5076d4d1b1c94612a
Jun 19 16:48:03 dist-test-slave-wvz7 krb5kdc[19672](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887683, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 16:48:03.658985 19908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:48:03.642577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:60449 (local address 127.0.97.62:34359)
0619 16:48:03.642752 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0619 16:48:03.642757 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:48:03.643880 (+ 1123us) server_negotiation.cc:407] Connection header received
0619 16:48:03.645772 (+ 1892us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:48:03.645778 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:48:03.645855 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:48:03.645943 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:48:03.647491 (+ 1548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:48:03.648295 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:48:03.650215 (+ 1920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:48:03.650633 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:48:03.650751 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:48:03.653980 (+ 3229us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:48:03.653985 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:48:03.653988 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:48:03.654018 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:48:03.656193 (+ 2175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:48:03.656827 (+ 634us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:48:03.656833 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:48:03.656836 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:48:03.656951 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:48:03.657346 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:48:03.657351 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:48:03.657355 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:48:03.657617 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:48:03.657782 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0619 16:48:03.658433 (+ 651us) server_negotiation.cc:299] Negotiation successful
0619 16:48:03.658642 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260619 16:48:03.660090 20175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:48:03.642996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34359 (local address 127.0.97.3:60449)
0619 16:48:03.643704 (+ 708us) negotiation.cc:107] Waiting for socket to connect
0619 16:48:03.643743 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0619 16:48:03.645460 (+ 1717us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:48:03.646191 (+ 731us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:48:03.646200 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:48:03.646760 (+ 560us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:48:03.647323 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:48:03.647339 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:48:03.648435 (+ 1096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:48:03.648439 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:48:03.650077 (+ 1638us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:48:03.650098 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:48:03.650836 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:48:03.650844 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:48:03.651087 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:48:03.651650 (+ 563us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:48:03.651671 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:48:03.653778 (+ 2107us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:48:03.656317 (+ 2539us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:48:03.656326 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:48:03.656342 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:48:03.656723 (+ 381us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:48:03.657057 (+ 334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:48:03.657062 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:48:03.657066 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:48:03.657246 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:48:03.657755 (+ 509us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:48:03.657763 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:48:03.658078 (+ 315us) client_negotiation.cc:769] Sending connection context
0619 16:48:03.658311 (+ 233us) client_negotiation.cc:240] Negotiation successful
0619 16:48:03.658793 (+ 482us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":481,"thread_start_us":295,"threads_started":1}
I20260619 16:48:03.661437 20173 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34359
I20260619 16:48:03.661717 20173 heartbeater.cc:464] Registering TS with master...
I20260619 16:48:03.662539 20173 heartbeater.cc:511] Master 127.0.97.62:34359 requested a full tablet report, sending...
I20260619 16:48:03.663905 19717 ts_manager.cc:194] Registered new tserver with Master: 0691559030a44924b787b5ed0bfe5dd3 (127.0.97.3:35767)
I20260619 16:48:03.664659 19717 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:60449
I20260619 16:48:03.675500 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 16:48:03.690953 19908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:48:03.679258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51962 (local address 127.0.97.62:34359)
0619 16:48:03.679403 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0619 16:48:03.679408 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:48:03.679868 (+ 460us) server_negotiation.cc:407] Connection header received
0619 16:48:03.679992 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:48:03.679996 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:48:03.680059 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:48:03.680139 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:48:03.681524 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:48:03.682328 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:48:03.683546 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:48:03.683960 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:48:03.684045 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:48:03.686197 (+ 2152us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:48:03.686202 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:48:03.686205 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:48:03.686265 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:48:03.688728 (+ 2463us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:48:03.689322 (+ 594us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:48:03.689326 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:48:03.689328 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:48:03.689441 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:48:03.689748 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:48:03.689754 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:48:03.689757 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:48:03.690073 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:48:03.690265 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0619 16:48:03.690509 (+ 244us) server_negotiation.cc:299] Negotiation successful
0619 16:48:03.690687 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260619 16:48:03.699597 19906 heartbeater.cc:503] Master 127.0.97.62:34359 was elected leader, sending a full tablet report...
I20260619 16:48:03.694693 19717 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51962:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 16:48:03.701514 19717 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 16:48:03.734817 20187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:48:03.713435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57528 (local address 127.0.97.3:35767)
0619 16:48:03.727819 (+ 14384us) server_negotiation.cc:206] Beginning negotiation
0619 16:48:03.727838 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0619 16:48:03.727868 (+ 30us) server_negotiation.cc:407] Connection header received
0619 16:48:03.727951 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:48:03.727958 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:48:03.728148 (+ 190us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:48:03.728321 (+ 173us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:48:03.728933 (+ 612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:48:03.730143 (+ 1210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:48:03.732833 (+ 2690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:48:03.733727 (+ 894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:48:03.733872 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:48:03.734104 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0619 16:48:03.734341 (+ 237us) server_negotiation.cc:299] Negotiation successful
0619 16:48:03.734501 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14220,"thread_start_us":14082,"threads_started":1}
I20260619 16:48:03.734982 20189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:48:03.712964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:35767 (local address 127.0.0.1:57528)
0619 16:48:03.714448 (+ 1484us) negotiation.cc:107] Waiting for socket to connect
0619 16:48:03.714465 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 16:48:03.714569 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:48:03.728333 (+ 13764us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:48:03.728337 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:48:03.728362 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:48:03.728767 (+ 405us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:48:03.728776 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:48:03.730288 (+ 1512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:48:03.730293 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:48:03.732645 (+ 2352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:48:03.732659 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:48:03.733916 (+ 1257us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:48:03.733922 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:48:03.734096 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:48:03.734146 (+ 50us) client_negotiation.cc:769] Sending connection context
0619 16:48:03.734298 (+ 152us) client_negotiation.cc:240] Negotiation successful
0619 16:48:03.734504 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1346,"mutex_wait_us":97,"thread_start_us":13926,"threads_started":1}
I20260619 16:48:03.735428 20185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:48:03.712645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:34093 (local address 127.0.0.1:53294)
0619 16:48:03.713499 (+ 854us) negotiation.cc:107] Waiting for socket to connect
0619 16:48:03.713546 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0619 16:48:03.713762 (+ 216us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:48:03.727977 (+ 14215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:48:03.727983 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:48:03.728014 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:48:03.728507 (+ 493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:48:03.728518 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:48:03.730090 (+ 1572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:48:03.730095 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:48:03.732648 (+ 2553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:48:03.732659 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:48:03.734376 (+ 1717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:48:03.734382 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:48:03.734550 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:48:03.734582 (+ 32us) client_negotiation.cc:769] Sending connection context
0619 16:48:03.734651 (+ 69us) client_negotiation.cc:240] Negotiation successful
0619 16:48:03.734778 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":598,"mutex_wait_us":127,"thread_start_us":421,"threads_started":1}
I20260619 16:48:03.735514 20188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:48:03.713925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:44715 (local address 127.0.0.1:37650)
0619 16:48:03.727135 (+ 13210us) negotiation.cc:107] Waiting for socket to connect
0619 16:48:03.727155 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 16:48:03.727281 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:48:03.728179 (+ 898us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:48:03.728184 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:48:03.728213 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:48:03.729128 (+ 915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:48:03.729139 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:48:03.730896 (+ 1757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:48:03.730900 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:48:03.732481 (+ 1581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:48:03.732496 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:48:03.734694 (+ 2198us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:48:03.734700 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:48:03.734868 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:48:03.734918 (+ 50us) client_negotiation.cc:769] Sending connection context
0619 16:48:03.735003 (+ 85us) client_negotiation.cc:240] Negotiation successful
0619 16:48:03.735118 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13094,"mutex_wait_us":1,"thread_start_us":618,"threads_started":1}
I20260619 16:48:03.735754 20186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:48:03.713298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53294 (local address 127.0.97.1:34093)
0619 16:48:03.727341 (+ 14043us) server_negotiation.cc:206] Beginning negotiation
0619 16:48:03.727350 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 16:48:03.727380 (+ 30us) server_negotiation.cc:407] Connection header received
0619 16:48:03.727489 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:48:03.727497 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:48:03.727757 (+ 260us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:48:03.727951 (+ 194us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:48:03.728658 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:48:03.729963 (+ 1305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:48:03.732801 (+ 2838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:48:03.734146 (+ 1345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:48:03.735016 (+ 870us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:48:03.735247 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0619 16:48:03.735325 (+ 78us) server_negotiation.cc:299] Negotiation successful
0619 16:48:03.735477 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13872,"thread_start_us":13763,"threads_started":1}
I20260619 16:48:03.737066 20190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:48:03.714390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37650 (local address 127.0.97.2:44715)
0619 16:48:03.726813 (+ 12423us) server_negotiation.cc:206] Beginning negotiation
0619 16:48:03.726824 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 16:48:03.727352 (+ 528us) server_negotiation.cc:407] Connection header received
0619 16:48:03.727470 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:48:03.727479 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:48:03.727763 (+ 284us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:48:03.727951 (+ 188us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:48:03.729290 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:48:03.730724 (+ 1434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:48:03.732627 (+ 1903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:48:03.734210 (+ 1583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:48:03.734364 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:48:03.734549 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0619 16:48:03.735632 (+ 1083us) server_negotiation.cc:299] Negotiation successful
0619 16:48:03.735800 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12208,"thread_start_us":12090,"threads_started":1}
I20260619 16:48:03.739269 20108 tablet_service.cc:1467] Processing CreateTablet for tablet 7bb524eb319e4c46b341daaf2e406fa0 (DEFAULT_TABLE table=test-table [id=6063993dcbbc4a88a1e83760c89d0ae4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:48:03.739461 19841 tablet_service.cc:1467] Processing CreateTablet for tablet 7bb524eb319e4c46b341daaf2e406fa0 (DEFAULT_TABLE table=test-table [id=6063993dcbbc4a88a1e83760c89d0ae4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:48:03.740463 20108 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7bb524eb319e4c46b341daaf2e406fa0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:48:03.740236 19975 tablet_service.cc:1467] Processing CreateTablet for tablet 7bb524eb319e4c46b341daaf2e406fa0 (DEFAULT_TABLE table=test-table [id=6063993dcbbc4a88a1e83760c89d0ae4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:48:03.740823 19841 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7bb524eb319e4c46b341daaf2e406fa0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:48:03.741470 19975 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7bb524eb319e4c46b341daaf2e406fa0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:48:03.751263 20191 tablet_bootstrap.cc:492] T 7bb524eb319e4c46b341daaf2e406fa0 P 0691559030a44924b787b5ed0bfe5dd3: Bootstrap starting.
I20260619 16:48:03.754741 20192 tablet_bootstrap.cc:492] T 7bb524eb319e4c46b341daaf2e406fa0 P 95f90b972b354f9a9c09b2b2432b0127: Bootstrap starting.
I20260619 16:48:03.755501 20193 tablet_bootstrap.cc:492] T 7bb524eb319e4c46b341daaf2e406fa0 P 7acfaf949855430b82f50146a5fba04f: Bootstrap starting.
I20260619 16:48:03.755563 20191 tablet_bootstrap.cc:654] T 7bb524eb319e4c46b341daaf2e406fa0 P 0691559030a44924b787b5ed0bfe5dd3: Neither blocks nor log segments found. Creating new log.
I20260619 16:48:03.756837 20191 log.cc:824] T 7bb524eb319e4c46b341daaf2e406fa0 P 0691559030a44924b787b5ed0bfe5dd3: Log is configured to *not* fsync() on all Append() calls
I20260619 16:48:03.758870 20193 tablet_bootstrap.cc:654] T 7bb524eb319e4c46b341daaf2e406fa0 P 7acfaf949855430b82f50146a5fba04f: Neither blocks nor log segments found. Creating new log.
I20260619 16:48:03.759361 20191 tablet_bootstrap.cc:492] T 7bb524eb319e4c46b341daaf2e406fa0 P 0691559030a44924b787b5ed0bfe5dd3: No bootstrap required, opened a new log
I20260619 16:48:03.759606 20191 ts_tablet_manager.cc:1397] T 7bb524eb319e4c46b341daaf2e406fa0 P 0691559030a44924b787b5ed0bfe5dd3: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260619 16:48:03.760017 20193 log.cc:824] T 7bb524eb319e4c46b341daaf2e406fa0 P 7acfaf949855430b82f50146a5fba04f: Log is configured to *not* fsync() on all Append() calls
I20260619 16:48:03.760861 20192 tablet_bootstrap.cc:654] T 7bb524eb319e4c46b341daaf2e406fa0 P 95f90b972b354f9a9c09b2b2432b0127: Neither blocks nor log segments found. Creating new log.
I20260619 16:48:03.761983 20192 log.cc:824] T 7bb524eb319e4c46b341daaf2e406fa0 P 95f90b972b354f9a9c09b2b2432b0127: Log is configured to *not* fsync() on all Append() calls
I20260619 16:48:03.762955 20193 tablet_bootstrap.cc:492] T 7bb524eb319e4c46b341daaf2e406fa0 P 7acfaf949855430b82f50146a5fba04f: No bootstrap required, opened a new log
I20260619 16:48:03.763289 20193 ts_tablet_manager.cc:1397] T 7bb524eb319e4c46b341daaf2e406fa0 P 7acfaf949855430b82f50146a5fba04f: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260619 16:48:03.765672 20192 tablet_bootstrap.cc:492] T 7bb524eb319e4c46b341daaf2e406fa0 P 95f90b972b354f9a9c09b2b2432b0127: No bootstrap required, opened a new log
I20260619 16:48:03.765939 20192 ts_tablet_manager.cc:1397] T 7bb524eb319e4c46b341daaf2e406fa0 P 95f90b972b354f9a9c09b2b2432b0127: Time spent bootstrapping tablet: real 0.011s user 0.008s sys 0.000s
I20260619 16:48:03.769536 20191 raft_consensus.cc:348] T 7bb524eb319e4c46b341daaf2e406fa0 P 0691559030a44924b787b5ed0bfe5dd3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95f90b972b354f9a9c09b2b2432b0127" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34093 } } peers { permanent_uuid: "0691559030a44924b787b5ed0bfe5dd3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35767 } } peers { permanent_uuid: "7acfaf949855430b82f50146a5fba04f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44715 } }
I20260619 16:48:03.770010 20191 raft_consensus.cc:374] T 7bb524eb319e4c46b341daaf2e406fa0 P 0691559030a44924b787b5ed0bfe5dd3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:48:03.770148 20191 raft_consensus.cc:729] T 7bb524eb319e4c46b341daaf2e406fa0 P 0691559030a44924b787b5ed0bfe5dd3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0691559030a44924b787b5ed0bfe5dd3, State: Initialized, Role: FOLLOWER
I20260619 16:48:03.770895 20191 consensus_queue.cc:260] T 7bb524eb319e4c46b341daaf2e406fa0 P 0691559030a44924b787b5ed0bfe5dd3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95f90b972b354f9a9c09b2b2432b0127" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34093 } } peers { permanent_uuid: "0691559030a44924b787b5ed0bfe5dd3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35767 } } peers { permanent_uuid: "7acfaf949855430b82f50146a5fba04f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44715 } }
I20260619 16:48:03.771962 20173 heartbeater.cc:503] Master 127.0.97.62:34359 was elected leader, sending a full tablet report...
I20260619 16:48:03.772641 20191 ts_tablet_manager.cc:1428] T 7bb524eb319e4c46b341daaf2e406fa0 P 0691559030a44924b787b5ed0bfe5dd3: Time spent starting tablet: real 0.013s user 0.013s sys 0.000s
I20260619 16:48:03.773136 20193 raft_consensus.cc:348] T 7bb524eb319e4c46b341daaf2e406fa0 P 7acfaf949855430b82f50146a5fba04f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95f90b972b354f9a9c09b2b2432b0127" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34093 } } peers { permanent_uuid: "0691559030a44924b787b5ed0bfe5dd3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35767 } } peers { permanent_uuid: "7acfaf949855430b82f50146a5fba04f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44715 } }
I20260619 16:48:03.773550 20193 raft_consensus.cc:374] T 7bb524eb319e4c46b341daaf2e406fa0 P 7acfaf949855430b82f50146a5fba04f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:48:03.773645 20193 raft_consensus.cc:729] T 7bb524eb319e4c46b341daaf2e406fa0 P 7acfaf949855430b82f50146a5fba04f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7acfaf949855430b82f50146a5fba04f, State: Initialized, Role: FOLLOWER
I20260619 16:48:03.774179 20193 consensus_queue.cc:260] T 7bb524eb319e4c46b341daaf2e406fa0 P 7acfaf949855430b82f50146a5fba04f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95f90b972b354f9a9c09b2b2432b0127" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34093 } } peers { permanent_uuid: "0691559030a44924b787b5ed0bfe5dd3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35767 } } peers { permanent_uuid: "7acfaf949855430b82f50146a5fba04f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44715 } }
I20260619 16:48:03.774434 20192 raft_consensus.cc:348] T 7bb524eb319e4c46b341daaf2e406fa0 P 95f90b972b354f9a9c09b2b2432b0127 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95f90b972b354f9a9c09b2b2432b0127" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34093 } } peers { permanent_uuid: "0691559030a44924b787b5ed0bfe5dd3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35767 } } peers { permanent_uuid: "7acfaf949855430b82f50146a5fba04f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44715 } }
I20260619 16:48:03.774883 20192 raft_consensus.cc:374] T 7bb524eb319e4c46b341daaf2e406fa0 P 95f90b972b354f9a9c09b2b2432b0127 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:48:03.775038 20192 raft_consensus.cc:729] T 7bb524eb319e4c46b341daaf2e406fa0 P 95f90b972b354f9a9c09b2b2432b0127 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 95f90b972b354f9a9c09b2b2432b0127, State: Initialized, Role: FOLLOWER
I20260619 16:48:03.775700 20193 ts_tablet_manager.cc:1428] T 7bb524eb319e4c46b341daaf2e406fa0 P 7acfaf949855430b82f50146a5fba04f: Time spent starting tablet: real 0.012s user 0.011s sys 0.000s
I20260619 16:48:03.775978 20040 heartbeater.cc:503] Master 127.0.97.62:34359 was elected leader, sending a full tablet report...
I20260619 16:48:03.776062 20192 consensus_queue.cc:260] T 7bb524eb319e4c46b341daaf2e406fa0 P 95f90b972b354f9a9c09b2b2432b0127 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95f90b972b354f9a9c09b2b2432b0127" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34093 } } peers { permanent_uuid: "0691559030a44924b787b5ed0bfe5dd3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35767 } } peers { permanent_uuid: "7acfaf949855430b82f50146a5fba04f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44715 } }
I20260619 16:48:03.779114 20192 ts_tablet_manager.cc:1428] T 7bb524eb319e4c46b341daaf2e406fa0 P 95f90b972b354f9a9c09b2b2432b0127: Time spent starting tablet: real 0.013s user 0.009s sys 0.001s
I20260619 16:48:03.785887 20199 raft_consensus.cc:482] T 7bb524eb319e4c46b341daaf2e406fa0 P 95f90b972b354f9a9c09b2b2432b0127 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 16:48:03.786098 20199 raft_consensus.cc:504] T 7bb524eb319e4c46b341daaf2e406fa0 P 95f90b972b354f9a9c09b2b2432b0127 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95f90b972b354f9a9c09b2b2432b0127" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34093 } } peers { permanent_uuid: "0691559030a44924b787b5ed0bfe5dd3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35767 } } peers { permanent_uuid: "7acfaf949855430b82f50146a5fba04f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44715 } }
I20260619 16:48:03.787562 20199 leader_election.cc:288] T 7bb524eb319e4c46b341daaf2e406fa0 P 95f90b972b354f9a9c09b2b2432b0127 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0691559030a44924b787b5ed0bfe5dd3 (127.0.97.3:35767), 7acfaf949855430b82f50146a5fba04f (127.0.97.2:44715)
I20260619 16:48:03.795663 20190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:48:03.787961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:46725 (local address 127.0.97.2:44715)
0619 16:48:03.788147 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0619 16:48:03.788153 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:48:03.788638 (+ 485us) server_negotiation.cc:407] Connection header received
0619 16:48:03.788756 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:48:03.788761 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:48:03.788853 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:48:03.788930 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:48:03.789687 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:48:03.790781 (+ 1094us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:48:03.793422 (+ 2641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:48:03.794656 (+ 1234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:48:03.794767 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:48:03.794874 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0619 16:48:03.795274 (+ 400us) server_negotiation.cc:299] Negotiation successful
0619 16:48:03.795403 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260619 16:48:03.795666 20187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:48:03.787961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:46063 (local address 127.0.97.3:35767)
0619 16:48:03.788249 (+ 288us) server_negotiation.cc:206] Beginning negotiation
0619 16:48:03.788254 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:48:03.788567 (+ 313us) server_negotiation.cc:407] Connection header received
0619 16:48:03.788756 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:48:03.788761 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:48:03.788834 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:48:03.788925 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:48:03.789687 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:48:03.790782 (+ 1095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:48:03.793425 (+ 2643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:48:03.794641 (+ 1216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:48:03.794768 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:48:03.794867 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0619 16:48:03.795330 (+ 463us) server_negotiation.cc:299] Negotiation successful
0619 16:48:03.795448 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":154}
I20260619 16:48:03.795905 20200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:48:03.787777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:35767 (local address 127.0.97.1:46063)
0619 16:48:03.788469 (+ 692us) negotiation.cc:107] Waiting for socket to connect
0619 16:48:03.788487 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 16:48:03.788631 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:48:03.788958 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:48:03.788963 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:48:03.789039 (+ 76us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:48:03.789573 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:48:03.789583 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:48:03.790901 (+ 1318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:48:03.790905 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:48:03.793276 (+ 2371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:48:03.793292 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:48:03.794883 (+ 1591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:48:03.794888 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:48:03.795095 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:48:03.795179 (+ 84us) client_negotiation.cc:769] Sending connection context
0619 16:48:03.795288 (+ 109us) client_negotiation.cc:240] Negotiation successful
0619 16:48:03.795420 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":552,"mutex_wait_us":140,"spinlock_wait_cycles":19712,"thread_start_us":345,"threads_started":1}
I20260619 16:48:03.796042 20201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:48:03.787863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:44715 (local address 127.0.97.1:46725)
0619 16:48:03.788551 (+ 688us) negotiation.cc:107] Waiting for socket to connect
0619 16:48:03.788565 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 16:48:03.788653 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:48:03.788958 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:48:03.788963 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:48:03.788990 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:48:03.789526 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:48:03.789543 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:48:03.790900 (+ 1357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:48:03.790905 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:48:03.793285 (+ 2380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:48:03.793293 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:48:03.794860 (+ 1567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:48:03.794872 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:48:03.795093 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:48:03.795155 (+ 62us) client_negotiation.cc:769] Sending connection context
0619 16:48:03.795259 (+ 104us) client_negotiation.cc:240] Negotiation successful
0619 16:48:03.795484 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":590,"mutex_wait_us":83,"thread_start_us":436,"threads_started":1}
I20260619 16:48:03.796748 20128 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7bb524eb319e4c46b341daaf2e406fa0" candidate_uuid: "95f90b972b354f9a9c09b2b2432b0127" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0691559030a44924b787b5ed0bfe5dd3" is_pre_election: true
I20260619 16:48:03.796841 19995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7bb524eb319e4c46b341daaf2e406fa0" candidate_uuid: "95f90b972b354f9a9c09b2b2432b0127" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7acfaf949855430b82f50146a5fba04f" is_pre_election: true
I20260619 16:48:03.797204 20128 raft_consensus.cc:2454] T 7bb524eb319e4c46b341daaf2e406fa0 P 0691559030a44924b787b5ed0bfe5dd3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 95f90b972b354f9a9c09b2b2432b0127 in term 0.
I20260619 16:48:03.797235 19995 raft_consensus.cc:2454] T 7bb524eb319e4c46b341daaf2e406fa0 P 7acfaf949855430b82f50146a5fba04f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 95f90b972b354f9a9c09b2b2432b0127 in term 0.
I20260619 16:48:03.797837 19795 leader_election.cc:302] T 7bb524eb319e4c46b341daaf2e406fa0 P 95f90b972b354f9a9c09b2b2432b0127 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 0691559030a44924b787b5ed0bfe5dd3, 7acfaf949855430b82f50146a5fba04f, 95f90b972b354f9a9c09b2b2432b0127; no voters:
I20260619 16:48:03.798202 20199 raft_consensus.cc:2781] T 7bb524eb319e4c46b341daaf2e406fa0 P 95f90b972b354f9a9c09b2b2432b0127 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 16:48:03.798321 20199 raft_consensus.cc:482] T 7bb524eb319e4c46b341daaf2e406fa0 P 95f90b972b354f9a9c09b2b2432b0127 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 16:48:03.798413 20199 raft_consensus.cc:3037] T 7bb524eb319e4c46b341daaf2e406fa0 P 95f90b972b354f9a9c09b2b2432b0127 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:48:03.800462 20199 raft_consensus.cc:504] T 7bb524eb319e4c46b341daaf2e406fa0 P 95f90b972b354f9a9c09b2b2432b0127 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95f90b972b354f9a9c09b2b2432b0127" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34093 } } peers { permanent_uuid: "0691559030a44924b787b5ed0bfe5dd3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35767 } } peers { permanent_uuid: "7acfaf949855430b82f50146a5fba04f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44715 } }
I20260619 16:48:03.801028 20199 leader_election.cc:288] T 7bb524eb319e4c46b341daaf2e406fa0 P 95f90b972b354f9a9c09b2b2432b0127 [CANDIDATE]: Term 1 election: Requested vote from peers 0691559030a44924b787b5ed0bfe5dd3 (127.0.97.3:35767), 7acfaf949855430b82f50146a5fba04f (127.0.97.2:44715)
I20260619 16:48:03.801468 20128 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7bb524eb319e4c46b341daaf2e406fa0" candidate_uuid: "95f90b972b354f9a9c09b2b2432b0127" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0691559030a44924b787b5ed0bfe5dd3"
I20260619 16:48:03.801472 19995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7bb524eb319e4c46b341daaf2e406fa0" candidate_uuid: "95f90b972b354f9a9c09b2b2432b0127" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7acfaf949855430b82f50146a5fba04f"
I20260619 16:48:03.801626 20128 raft_consensus.cc:3037] T 7bb524eb319e4c46b341daaf2e406fa0 P 0691559030a44924b787b5ed0bfe5dd3 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:48:03.801673 19995 raft_consensus.cc:3037] T 7bb524eb319e4c46b341daaf2e406fa0 P 7acfaf949855430b82f50146a5fba04f [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:48:03.803714 20128 raft_consensus.cc:2454] T 7bb524eb319e4c46b341daaf2e406fa0 P 0691559030a44924b787b5ed0bfe5dd3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 95f90b972b354f9a9c09b2b2432b0127 in term 1.
I20260619 16:48:03.804013 19995 raft_consensus.cc:2454] T 7bb524eb319e4c46b341daaf2e406fa0 P 7acfaf949855430b82f50146a5fba04f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 95f90b972b354f9a9c09b2b2432b0127 in term 1.
I20260619 16:48:03.804064 19795 leader_election.cc:302] T 7bb524eb319e4c46b341daaf2e406fa0 P 95f90b972b354f9a9c09b2b2432b0127 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0691559030a44924b787b5ed0bfe5dd3, 95f90b972b354f9a9c09b2b2432b0127; no voters:
I20260619 16:48:03.804388 20199 raft_consensus.cc:2781] T 7bb524eb319e4c46b341daaf2e406fa0 P 95f90b972b354f9a9c09b2b2432b0127 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:48:03.804839 20199 raft_consensus.cc:686] T 7bb524eb319e4c46b341daaf2e406fa0 P 95f90b972b354f9a9c09b2b2432b0127 [term 1 LEADER]: Becoming Leader. State: Replica: 95f90b972b354f9a9c09b2b2432b0127, State: Running, Role: LEADER
I20260619 16:48:03.805300 20199 consensus_queue.cc:237] T 7bb524eb319e4c46b341daaf2e406fa0 P 95f90b972b354f9a9c09b2b2432b0127 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95f90b972b354f9a9c09b2b2432b0127" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34093 } } peers { permanent_uuid: "0691559030a44924b787b5ed0bfe5dd3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35767 } } peers { permanent_uuid: "7acfaf949855430b82f50146a5fba04f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44715 } }
I20260619 16:48:03.809795 19716 catalog_manager.cc:5495] T 7bb524eb319e4c46b341daaf2e406fa0 P 95f90b972b354f9a9c09b2b2432b0127 reported cstate change: term changed from 0 to 1, leader changed from <none> to 95f90b972b354f9a9c09b2b2432b0127 (127.0.97.1). New cstate: current_term: 1 leader_uuid: "95f90b972b354f9a9c09b2b2432b0127" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95f90b972b354f9a9c09b2b2432b0127" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0691559030a44924b787b5ed0bfe5dd3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35767 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7acfaf949855430b82f50146a5fba04f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44715 } health_report { overall_health: UNKNOWN } } }
I20260619 16:48:03.847244 20186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:48:03.842266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53296 (local address 127.0.97.1:34093)
0619 16:48:03.842504 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0619 16:48:03.842510 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:48:03.842530 (+ 20us) server_negotiation.cc:407] Connection header received
0619 16:48:03.842651 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:48:03.842655 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:48:03.842734 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:48:03.842856 (+ 122us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 16:48:03.843138 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:48:03.843851 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:48:03.845225 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:48:03.845626 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:48:03.845717 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:48:03.846222 (+ 505us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 16:48:03.846700 (+ 478us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 16:48:03.846826 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0619 16:48:03.846922 (+ 96us) server_negotiation.cc:299] Negotiation successful
0619 16:48:03.846988 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260619 16:48:03.855110 19995 raft_consensus.cc:1261] T 7bb524eb319e4c46b341daaf2e406fa0 P 7acfaf949855430b82f50146a5fba04f [term 1 FOLLOWER]: Refusing update from remote peer 95f90b972b354f9a9c09b2b2432b0127: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:48:03.855109 20128 raft_consensus.cc:1261] T 7bb524eb319e4c46b341daaf2e406fa0 P 0691559030a44924b787b5ed0bfe5dd3 [term 1 FOLLOWER]: Refusing update from remote peer 95f90b972b354f9a9c09b2b2432b0127: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:48:03.855980 20202 consensus_queue.cc:1035] T 7bb524eb319e4c46b341daaf2e406fa0 P 95f90b972b354f9a9c09b2b2432b0127 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0691559030a44924b787b5ed0bfe5dd3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:48:03.856290 20199 consensus_queue.cc:1035] T 7bb524eb319e4c46b341daaf2e406fa0 P 95f90b972b354f9a9c09b2b2432b0127 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7acfaf949855430b82f50146a5fba04f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44715 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:48:03.867786 20205 mvcc.cc:205] Tried to move back new op lower bound from 7298611953061138432 to 7298611952870612992. Current Snapshot: MvccSnapshot[applied={T|T < 7298611953061138432}]
I20260619 16:48:03.878427 19717 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51962:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:48:03.878682 19717 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51962:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:48:03.881978 19717 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a8f4fe346a4c4a03a94b2e779e3daef0: Sending DeleteTablet for 3 replicas of tablet 7bb524eb319e4c46b341daaf2e406fa0
I20260619 16:48:03.883118 19975 tablet_service.cc:1514] Processing DeleteTablet for tablet 7bb524eb319e4c46b341daaf2e406fa0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:48:03 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:37650
I20260619 16:48:03.883107 20108 tablet_service.cc:1514] Processing DeleteTablet for tablet 7bb524eb319e4c46b341daaf2e406fa0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:48:03 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:57528
I20260619 16:48:03.883248 19841 tablet_service.cc:1514] Processing DeleteTablet for tablet 7bb524eb319e4c46b341daaf2e406fa0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:48:03 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:53294
I20260619 16:48:03.885385 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu pid 19778
I20260619 16:48:03.902624 20215 tablet_replica.cc:330] T 7bb524eb319e4c46b341daaf2e406fa0 P 7acfaf949855430b82f50146a5fba04f: stopping tablet replica
I20260619 16:48:03.903330 20215 raft_consensus.cc:2229] T 7bb524eb319e4c46b341daaf2e406fa0 P 7acfaf949855430b82f50146a5fba04f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 16:48:03.904039 20215 raft_consensus.cc:2258] T 7bb524eb319e4c46b341daaf2e406fa0 P 7acfaf949855430b82f50146a5fba04f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:48:03.905113 20214 tablet_replica.cc:330] T 7bb524eb319e4c46b341daaf2e406fa0 P 0691559030a44924b787b5ed0bfe5dd3: stopping tablet replica
I20260619 16:48:03.906695 20215 ts_tablet_manager.cc:1905] T 7bb524eb319e4c46b341daaf2e406fa0 P 7acfaf949855430b82f50146a5fba04f: Deleting tablet data with delete state TABLET_DATA_DELETED
Jun 19 16:48:03 dist-test-slave-wvz7 krb5kdc[19672](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887681, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260619 16:48:03.913640 20215 ts_tablet_manager.cc:1918] T 7bb524eb319e4c46b341daaf2e406fa0 P 7acfaf949855430b82f50146a5fba04f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 16:48:03.914333 20216 tablet_replica.cc:330] T 7bb524eb319e4c46b341daaf2e406fa0 P 95f90b972b354f9a9c09b2b2432b0127: stopping tablet replica
I20260619 16:48:03.914589 20215 log.cc:1196] T 7bb524eb319e4c46b341daaf2e406fa0 P 7acfaf949855430b82f50146a5fba04f: Deleting WAL directory at /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887385903558-388-0/minicluster-data/ts-1/wal/wals/7bb524eb319e4c46b341daaf2e406fa0
I20260619 16:48:03.914798 20216 raft_consensus.cc:2229] T 7bb524eb319e4c46b341daaf2e406fa0 P 95f90b972b354f9a9c09b2b2432b0127 [term 1 LEADER]: Raft consensus shutting down.
I20260619 16:48:03.915019 20214 raft_consensus.cc:2229] T 7bb524eb319e4c46b341daaf2e406fa0 P 0691559030a44924b787b5ed0bfe5dd3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 16:48:03.915225 20215 ts_tablet_manager.cc:1939] T 7bb524eb319e4c46b341daaf2e406fa0 P 7acfaf949855430b82f50146a5fba04f: Deleting consensus metadata
I20260619 16:48:03.915709 20216 raft_consensus.cc:2258] T 7bb524eb319e4c46b341daaf2e406fa0 P 95f90b972b354f9a9c09b2b2432b0127 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:48:03.916419 20214 raft_consensus.cc:2258] T 7bb524eb319e4c46b341daaf2e406fa0 P 0691559030a44924b787b5ed0bfe5dd3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:48:03.917164 19703 catalog_manager.cc:4841] TS 7acfaf949855430b82f50146a5fba04f (127.0.97.2:44715): tablet 7bb524eb319e4c46b341daaf2e406fa0 (table test-table [id=6063993dcbbc4a88a1e83760c89d0ae4]) successfully deleted
I20260619 16:48:03.918444 20216 ts_tablet_manager.cc:1905] T 7bb524eb319e4c46b341daaf2e406fa0 P 95f90b972b354f9a9c09b2b2432b0127: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:48:03.918766 20186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:48:03.890003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53304 (local address 127.0.97.1:34093)
0619 16:48:03.890531 (+ 528us) server_negotiation.cc:206] Beginning negotiation
0619 16:48:03.890538 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:48:03.892868 (+ 2330us) server_negotiation.cc:407] Connection header received
0619 16:48:03.892950 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:48:03.892956 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:48:03.893045 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:48:03.893153 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:48:03.895858 (+ 2705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:48:03.897189 (+ 1331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:48:03.901174 (+ 3985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:48:03.902030 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:48:03.902141 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:48:03.908694 (+ 6553us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:48:03.908710 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:48:03.908727 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:48:03.908778 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:48:03.912709 (+ 3931us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:48:03.915265 (+ 2556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:48:03.915274 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:48:03.915277 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:48:03.915491 (+ 214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:48:03.916736 (+ 1245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:48:03.916746 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:48:03.916749 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:48:03.917201 (+ 452us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:48:03.917471 (+ 270us) server_negotiation.cc:1035] Waiting for connection context
0619 16:48:03.918173 (+ 702us) server_negotiation.cc:299] Negotiation successful
0619 16:48:03.918437 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":392}
I20260619 16:48:03.919529 20214 ts_tablet_manager.cc:1905] T 7bb524eb319e4c46b341daaf2e406fa0 P 0691559030a44924b787b5ed0bfe5dd3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:48:03.920487 19901 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:48:03.925818 20214 ts_tablet_manager.cc:1918] T 7bb524eb319e4c46b341daaf2e406fa0 P 0691559030a44924b787b5ed0bfe5dd3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 16:48:03.925958 20214 log.cc:1196] T 7bb524eb319e4c46b341daaf2e406fa0 P 0691559030a44924b787b5ed0bfe5dd3: Deleting WAL directory at /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887385903558-388-0/minicluster-data/ts-2/wal/wals/7bb524eb319e4c46b341daaf2e406fa0
I20260619 16:48:03.926374 20214 ts_tablet_manager.cc:1939] T 7bb524eb319e4c46b341daaf2e406fa0 P 0691559030a44924b787b5ed0bfe5dd3: Deleting consensus metadata
I20260619 16:48:03.927852 19702 catalog_manager.cc:4841] TS 0691559030a44924b787b5ed0bfe5dd3 (127.0.97.3:35767): tablet 7bb524eb319e4c46b341daaf2e406fa0 (table test-table [id=6063993dcbbc4a88a1e83760c89d0ae4]) successfully deleted
I20260619 16:48:04.016187 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu with pid 19778
W20260619 16:48:04.030934 19703 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.97.1:34093 (error 108)
I20260619 16:48:04.031140 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu pid 19912
I20260619 16:48:04.032946 20188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:48:04.032444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:34093 (local address 127.0.0.1:53314)
0619 16:48:04.032619 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0619 16:48:04.032753 (+ 134us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:34093: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260619 16:48:04.033205 19703 catalog_manager.cc:4584] TS 95f90b972b354f9a9c09b2b2432b0127 (127.0.97.1:34093): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 7bb524eb319e4c46b341daaf2e406fa0: Network error: Client connection negotiation failed: client connection to 127.0.97.1:34093: connect: Connection refused (error 111)
Jun 19 16:48:04 dist-test-slave-wvz7 krb5kdc[19672](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887681, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260619 16:48:04.046324 20190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:48:04.031857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37656 (local address 127.0.97.2:44715)
0619 16:48:04.032013 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0619 16:48:04.032017 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:48:04.032090 (+ 73us) server_negotiation.cc:407] Connection header received
0619 16:48:04.032289 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:48:04.032294 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:48:04.032395 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:48:04.032502 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:48:04.033985 (+ 1483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:48:04.034744 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:48:04.036002 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:48:04.036377 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:48:04.036469 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:48:04.040752 (+ 4283us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:48:04.040767 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:48:04.040782 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:48:04.040840 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:48:04.043613 (+ 2773us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:48:04.044343 (+ 730us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:48:04.044351 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:48:04.044353 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:48:04.044482 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:48:04.044854 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:48:04.044857 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:48:04.044859 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:48:04.045230 (+ 371us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:48:04.045446 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0619 16:48:04.045829 (+ 383us) server_negotiation.cc:299] Negotiation successful
0619 16:48:04.046000 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260619 16:48:04.046963 20035 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:48:04.087002 20188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:48:04.086457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:34093 (local address 127.0.0.1:53326)
0619 16:48:04.086693 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0619 16:48:04.086815 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:34093: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20260619 16:48:04.088114 20188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:48:04.087739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:34093 (local address 127.0.0.1:53336)
0619 16:48:04.087889 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0619 16:48:04.087979 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:34093: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260619 16:48:04.136471 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu with pid 19912
I20260619 16:48:04.148967 20188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:48:04.148405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:34093 (local address 127.0.0.1:53338)
0619 16:48:04.148668 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0619 16:48:04.148771 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:34093: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99}
I20260619 16:48:04.149956 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu pid 20045
I20260619 16:48:04.150285 20188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:48:04.149899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:34093 (local address 127.0.0.1:53342)
0619 16:48:04.150080 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0619 16:48:04.150170 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:34093: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
Jun 19 16:48:04 dist-test-slave-wvz7 krb5kdc[19672](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887681, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260619 16:48:04.164141 20187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:48:04.150709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57542 (local address 127.0.97.3:35767)
0619 16:48:04.150887 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0619 16:48:04.150894 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:48:04.150912 (+ 18us) server_negotiation.cc:407] Connection header received
0619 16:48:04.151064 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:48:04.151070 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:48:04.151142 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:48:04.151231 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:48:04.152487 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:48:04.153215 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:48:04.154513 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:48:04.154929 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:48:04.155036 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:48:04.159052 (+ 4016us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:48:04.159068 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:48:04.159082 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:48:04.159115 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:48:04.161648 (+ 2533us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:48:04.162387 (+ 739us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:48:04.162397 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:48:04.162400 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:48:04.162526 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:48:04.162913 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:48:04.162916 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:48:04.162919 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:48:04.163245 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:48:04.163461 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0619 16:48:04.163708 (+ 247us) server_negotiation.cc:299] Negotiation successful
0619 16:48:04.163884 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260619 16:48:04.164692 20168 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:48:04.230867 20188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:48:04.230374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:34093 (local address 127.0.0.1:53344)
0619 16:48:04.230594 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0619 16:48:04.230689 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:34093: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20260619 16:48:04.231982 20188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:48:04.231577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:34093 (local address 127.0.0.1:53360)
0619 16:48:04.231797 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0619 16:48:04.231870 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:34093: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125}
I20260619 16:48:04.253947 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu with pid 20045
I20260619 16:48:04.267241 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu pid 19685
I20260619 16:48:04.268288 19747 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:48:04.378611 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu with pid 19685
2026-06-19T16:48:04Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (3985 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-06-19T16:48:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T16:48:04Z Disabled control of system clock
I20260619 16:48:04.447868 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu
/tmp/dist-test-taskEs2EH8/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781887385903558-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781887385903558-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781887385903558-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781887385903558-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35245
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:44289
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:35245
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 16:48:04.762799 20226 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:48:04.763078 20226 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:48:04.763191 20226 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:48:04.770498 20226 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:48:04.770607 20226 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:48:04.770653 20226 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:48:04.770730 20226 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:48:04.770783 20226 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:48:04.780123 20226 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44289
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781887385903558-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781887385903558-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:35245
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35245
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781887385903558-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781887385903558-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:48:04.781358 20226 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:48:04.783064 20226 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:48:04.793978 20231 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:48:04.793916 20232 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:48:04.793959 20234 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:48:04.795437 20226 server_base.cc:1034] running on GCE node
I20260619 16:48:04.796221 20226 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:48:04.797672 20226 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:48:04.798923 20226 hybrid_clock.cc:648] HybridClock initialized: now 1781887684798872 us; error 42 us; skew 500 ppm
I20260619 16:48:04.799284 20226 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 16:48:04.801637 20226 webserver.cc:460] Webserver started at http://127.0.97.62:38727/ using document root <none> and password file <none>
I20260619 16:48:04.802346 20226 fs_manager.cc:359] Metadata directory not provided
I20260619 16:48:04.802452 20226 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:48:04.802760 20226 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:48:04.805598 20226 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781887385903558-388-0/minicluster-data/master-0/data/instance:
uuid: "15a9096772d043a9b21a26c0971240ac"
format_stamp: "Formatted at 2026-06-19 16:48:04 on dist-test-slave-wvz7"
server_key: "5a9b3bf080e31d4ee2d4325309bf68a6"
server_key_iv: "fbf7f4714023506adea3808f9d819017"
server_key_version: "encryptionkey@0"
I20260619 16:48:04.806442 20226 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781887385903558-388-0/minicluster-data/master-0/wal/instance:
uuid: "15a9096772d043a9b21a26c0971240ac"
format_stamp: "Formatted at 2026-06-19 16:48:04 on dist-test-slave-wvz7"
server_key: "5a9b3bf080e31d4ee2d4325309bf68a6"
server_key_iv: "fbf7f4714023506adea3808f9d819017"
server_key_version: "encryptionkey@0"
I20260619 16:48:04.811514 20226 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 16:48:04.814940 20240 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:48:04.816249 20226 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 16:48:04.816417 20226 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781887385903558-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781887385903558-388-0/minicluster-data/master-0/wal
uuid: "15a9096772d043a9b21a26c0971240ac"
format_stamp: "Formatted at 2026-06-19 16:48:04 on dist-test-slave-wvz7"
server_key: "5a9b3bf080e31d4ee2d4325309bf68a6"
server_key_iv: "fbf7f4714023506adea3808f9d819017"
server_key_version: "encryptionkey@0"
I20260619 16:48:04.816608 20226 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781887385903558-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781887385903558-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781887385903558-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:48:04.835538 20226 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:48:04.836301 20226 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:48:04.836549 20226 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:48:04.853237 20226 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:35245
I20260619 16:48:04.853288 20291 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:35245 every 8 connection(s)
I20260619 16:48:04.854478 20226 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781887385903558-388-0/minicluster-data/master-0/data/info.pb
I20260619 16:48:04.858153 20292 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:48:04.858798 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu as pid 20226
I20260619 16:48:04.858952 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781887385903558-388-0/minicluster-data/master-0/wal/instance
I20260619 16:48:04.859344 388 external_mini_cluster.cc:1428] Setting key 70b111daaac93764c8fe18792395428c
I20260619 16:48:04.873072 20292 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 15a9096772d043a9b21a26c0971240ac: Bootstrap starting.
I20260619 16:48:04.875228 20295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:48:04.861182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34712 (local address 127.0.97.62:35245)
0619 16:48:04.862315 (+ 1133us) server_negotiation.cc:206] Beginning negotiation
0619 16:48:04.862334 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0619 16:48:04.862387 (+ 53us) server_negotiation.cc:407] Connection header received
0619 16:48:04.864902 (+ 2515us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:48:04.864932 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:48:04.865567 (+ 635us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:48:04.866068 (+ 501us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:48:04.866613 (+ 545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:48:04.868458 (+ 1845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:48:04.869820 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:48:04.870955 (+ 1135us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:48:04.871172 (+ 217us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:48:04.872274 (+ 1102us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:48:04.872286 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:48:04.872305 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:48:04.873055 (+ 750us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:48:04.873181 (+ 126us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:48:04.873212 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:48:04.873348 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0619 16:48:04.873514 (+ 166us) server_negotiation.cc:299] Negotiation successful
0619 16:48:04.873876 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":585,"thread_start_us":342,"threads_started":1}
I20260619 16:48:04.877216 20292 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 15a9096772d043a9b21a26c0971240ac: Neither blocks nor log segments found. Creating new log.
I20260619 16:48:04.878463 20292 log.cc:824] T 00000000000000000000000000000000 P 15a9096772d043a9b21a26c0971240ac: Log is configured to *not* fsync() on all Append() calls
I20260619 16:48:04.881103 20292 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 15a9096772d043a9b21a26c0971240ac: No bootstrap required, opened a new log
I20260619 16:48:04.888830 20292 raft_consensus.cc:348] T 00000000000000000000000000000000 P 15a9096772d043a9b21a26c0971240ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15a9096772d043a9b21a26c0971240ac" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35245 } }
I20260619 16:48:04.889186 20292 raft_consensus.cc:374] T 00000000000000000000000000000000 P 15a9096772d043a9b21a26c0971240ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:48:04.889294 20292 raft_consensus.cc:729] T 00000000000000000000000000000000 P 15a9096772d043a9b21a26c0971240ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 15a9096772d043a9b21a26c0971240ac, State: Initialized, Role: FOLLOWER
I20260619 16:48:04.889907 20292 consensus_queue.cc:260] T 00000000000000000000000000000000 P 15a9096772d043a9b21a26c0971240ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15a9096772d043a9b21a26c0971240ac" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35245 } }
I20260619 16:48:04.890144 20292 raft_consensus.cc:388] T 00000000000000000000000000000000 P 15a9096772d043a9b21a26c0971240ac [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:48:04.890286 20292 raft_consensus.cc:482] T 00000000000000000000000000000000 P 15a9096772d043a9b21a26c0971240ac [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:48:04.890442 20292 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 15a9096772d043a9b21a26c0971240ac [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:48:04.892850 20292 raft_consensus.cc:504] T 00000000000000000000000000000000 P 15a9096772d043a9b21a26c0971240ac [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15a9096772d043a9b21a26c0971240ac" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35245 } }
I20260619 16:48:04.893342 20292 leader_election.cc:302] T 00000000000000000000000000000000 P 15a9096772d043a9b21a26c0971240ac [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 15a9096772d043a9b21a26c0971240ac; no voters:
I20260619 16:48:04.893934 20292 leader_election.cc:288] T 00000000000000000000000000000000 P 15a9096772d043a9b21a26c0971240ac [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 16:48:04.894042 20297 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 15a9096772d043a9b21a26c0971240ac [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:48:04.894688 20297 raft_consensus.cc:686] T 00000000000000000000000000000000 P 15a9096772d043a9b21a26c0971240ac [term 1 LEADER]: Becoming Leader. State: Replica: 15a9096772d043a9b21a26c0971240ac, State: Running, Role: LEADER
I20260619 16:48:04.895169 20297 consensus_queue.cc:237] T 00000000000000000000000000000000 P 15a9096772d043a9b21a26c0971240ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15a9096772d043a9b21a26c0971240ac" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35245 } }
I20260619 16:48:04.895989 20292 sys_catalog.cc:564] T 00000000000000000000000000000000 P 15a9096772d043a9b21a26c0971240ac [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:48:04.897645 20298 sys_catalog.cc:455] T 00000000000000000000000000000000 P 15a9096772d043a9b21a26c0971240ac [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "15a9096772d043a9b21a26c0971240ac" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15a9096772d043a9b21a26c0971240ac" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35245 } } }
I20260619 16:48:04.898165 20298 sys_catalog.cc:458] T 00000000000000000000000000000000 P 15a9096772d043a9b21a26c0971240ac [sys.catalog]: This master's current role is: LEADER
I20260619 16:48:04.899004 20299 sys_catalog.cc:455] T 00000000000000000000000000000000 P 15a9096772d043a9b21a26c0971240ac [sys.catalog]: SysCatalogTable state changed. Reason: New leader 15a9096772d043a9b21a26c0971240ac. Latest consensus state: current_term: 1 leader_uuid: "15a9096772d043a9b21a26c0971240ac" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15a9096772d043a9b21a26c0971240ac" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35245 } } }
I20260619 16:48:04.899330 20299 sys_catalog.cc:458] T 00000000000000000000000000000000 P 15a9096772d043a9b21a26c0971240ac [sys.catalog]: This master's current role is: LEADER
I20260619 16:48:04.903108 20304 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:48:04.907402 20304 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:48:04.915916 20304 catalog_manager.cc:1347] Generated new cluster ID: 94a3a914f6a54f8ea6fd68fb67fb5c7e
I20260619 16:48:04.916013 20304 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:48:04.931541 20304 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 16:48:04.932760 20304 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:48:04.943202 20304 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 15a9096772d043a9b21a26c0971240ac: Generated new TSK 0
I20260619 16:48:04.944341 20304 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 16:48:04.958056 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu
/tmp/dist-test-taskEs2EH8/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781887385903558-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781887385903558-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781887385903558-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781887385903558-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35245
--builtin_ntp_servers=127.0.97.20:44289
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 16:48:05.278481 20316 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:48:05.278731 20316 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:48:05.278841 20316 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:48:05.285928 20316 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:48:05.286016 20316 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:48:05.286154 20316 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260619 16:48:05.294909 20316 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44289
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781887385903558-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781887385903558-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781887385903558-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35245
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781887385903558-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:48:05.296142 20316 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:48:05.298004 20316 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:48:05.310555 20321 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:48:05.310523 20322 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:48:05.311501 20324 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:48:05.312914 20316 server_base.cc:1034] running on GCE node
I20260619 16:48:05.313627 20316 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:48:05.314944 20316 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:48:05.316190 20316 hybrid_clock.cc:648] HybridClock initialized: now 1781887685316130 us; error 44 us; skew 500 ppm
I20260619 16:48:05.316566 20316 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 16:48:05.319309 20316 webserver.cc:460] Webserver started at http://127.0.97.1:41489/ using document root <none> and password file <none>
I20260619 16:48:05.320202 20316 fs_manager.cc:359] Metadata directory not provided
I20260619 16:48:05.320320 20316 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:48:05.320694 20316 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:48:05.324770 20316 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781887385903558-388-0/minicluster-data/ts-0/data/instance:
uuid: "336a948d93e44f3eb1fe2c395da95049"
format_stamp: "Formatted at 2026-06-19 16:48:05 on dist-test-slave-wvz7"
server_key: "157afc7b5f92ce5eb91b09f82365b44f"
server_key_iv: "2ce7bb4f6da371ddf7f26dc97bdcdc27"
server_key_version: "encryptionkey@0"
I20260619 16:48:05.325759 20316 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781887385903558-388-0/minicluster-data/ts-0/wal/instance:
uuid: "336a948d93e44f3eb1fe2c395da95049"
format_stamp: "Formatted at 2026-06-19 16:48:05 on dist-test-slave-wvz7"
server_key: "157afc7b5f92ce5eb91b09f82365b44f"
server_key_iv: "2ce7bb4f6da371ddf7f26dc97bdcdc27"
server_key_version: "encryptionkey@0"
I20260619 16:48:05.331735 20316 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260619 16:48:05.335897 20330 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:48:05.337585 20316 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 16:48:05.337834 20316 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781887385903558-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781887385903558-388-0/minicluster-data/ts-0/wal
uuid: "336a948d93e44f3eb1fe2c395da95049"
format_stamp: "Formatted at 2026-06-19 16:48:05 on dist-test-slave-wvz7"
server_key: "157afc7b5f92ce5eb91b09f82365b44f"
server_key_iv: "2ce7bb4f6da371ddf7f26dc97bdcdc27"
server_key_version: "encryptionkey@0"
I20260619 16:48:05.338090 20316 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781887385903558-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781887385903558-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781887385903558-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:48:05.364060 20316 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:48:05.364859 20316 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:48:05.365087 20316 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:48:05.366065 20316 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:48:05.367532 20316 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:48:05.367615 20316 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:48:05.367691 20316 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:48:05.367743 20316 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:48:05.397064 20316 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:43589
I20260619 16:48:05.397116 20442 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:43589 every 8 connection(s)
I20260619 16:48:05.398476 20316 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781887385903558-388-0/minicluster-data/ts-0/data/info.pb
I20260619 16:48:05.404295 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu as pid 20316
I20260619 16:48:05.404465 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781887385903558-388-0/minicluster-data/ts-0/wal/instance
I20260619 16:48:05.404971 388 external_mini_cluster.cc:1428] Setting key 3f50d65175b8e474933123d2094f9e65
I20260619 16:48:05.407868 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu
/tmp/dist-test-taskEs2EH8/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781887385903558-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781887385903558-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781887385903558-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781887385903558-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35245
--builtin_ntp_servers=127.0.97.20:44289
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260619 16:48:05.412470 20445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:48:05.401049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:34029 (local address 127.0.97.62:35245)
0619 16:48:05.401515 (+ 466us) server_negotiation.cc:206] Beginning negotiation
0619 16:48:05.401521 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:48:05.402741 (+ 1220us) server_negotiation.cc:407] Connection header received
0619 16:48:05.404844 (+ 2103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:48:05.404849 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:48:05.404919 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:48:05.405014 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:48:05.406469 (+ 1455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:48:05.407323 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:48:05.409806 (+ 2483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:48:05.410215 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:48:05.410323 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:48:05.411777 (+ 1454us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:48:05.411783 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:48:05.411787 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:48:05.411806 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:48:05.411844 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:48:05.411857 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:48:05.411925 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0619 16:48:05.412138 (+ 213us) server_negotiation.cc:299] Negotiation successful
0619 16:48:05.412218 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":335,"thread_start_us":212,"threads_started":1}
I20260619 16:48:05.413839 20446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:48:05.401653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35245 (local address 127.0.97.1:34029)
0619 16:48:05.402589 (+ 936us) negotiation.cc:107] Waiting for socket to connect
0619 16:48:05.402628 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0619 16:48:05.404522 (+ 1894us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:48:05.405284 (+ 762us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:48:05.405297 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:48:05.405422 (+ 125us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:48:05.406294 (+ 872us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:48:05.406314 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:48:05.407494 (+ 1180us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:48:05.407500 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:48:05.409585 (+ 2085us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:48:05.409609 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:48:05.410436 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:48:05.410443 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:48:05.410710 (+ 267us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:48:05.411422 (+ 712us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 16:48:05.411425 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:48:05.411476 (+ 51us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 16:48:05.411579 (+ 103us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:48:05.411941 (+ 362us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:48:05.411950 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:48:05.411957 (+ 7us) client_negotiation.cc:769] Sending connection context
0619 16:48:05.412131 (+ 174us) client_negotiation.cc:240] Negotiation successful
0619 16:48:05.412537 (+ 406us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":466,"thread_start_us":258,"threads_started":1}
I20260619 16:48:05.415410 20443 heartbeater.cc:347] Connected to a master server at 127.0.97.62:35245
I20260619 16:48:05.415726 20443 heartbeater.cc:464] Registering TS with master...
I20260619 16:48:05.416553 20443 heartbeater.cc:511] Master 127.0.97.62:35245 requested a full tablet report, sending...
I20260619 16:48:05.418712 20257 ts_manager.cc:194] Registered new tserver with Master: 336a948d93e44f3eb1fe2c395da95049 (127.0.97.1:43589)
I20260619 16:48:05.420655 20257 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.1:34029
W20260619 16:48:05.722746 20447 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:48:05.723011 20447 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:48:05.723119 20447 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:48:05.730264 20447 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:48:05.730360 20447 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:48:05.730505 20447 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260619 16:48:05.739332 20447 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44289
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781887385903558-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781887385903558-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781887385903558-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35245
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781887385903558-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:48:05.740573 20447 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:48:05.742352 20447 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:48:05.754961 20452 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:48:05.755039 20453 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:48:05.755002 20455 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:48:05.755743 20447 server_base.cc:1034] running on GCE node
I20260619 16:48:05.756338 20447 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:48:05.757261 20447 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:48:05.758453 20447 hybrid_clock.cc:648] HybridClock initialized: now 1781887685758413 us; error 31 us; skew 500 ppm
I20260619 16:48:05.758751 20447 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 16:48:05.763479 20447 webserver.cc:460] Webserver started at http://127.0.97.2:34159/ using document root <none> and password file <none>
I20260619 16:48:05.764158 20447 fs_manager.cc:359] Metadata directory not provided
I20260619 16:48:05.764254 20447 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:48:05.764560 20447 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:48:05.767345 20447 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781887385903558-388-0/minicluster-data/ts-1/data/instance:
uuid: "1b25dd24c7a04eca99d161491588e3ba"
format_stamp: "Formatted at 2026-06-19 16:48:05 on dist-test-slave-wvz7"
server_key: "57411f61cf28c87675835c41b612c140"
server_key_iv: "a8b942b5e19d4aa744c6ec717fc42663"
server_key_version: "encryptionkey@0"
I20260619 16:48:05.768085 20447 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781887385903558-388-0/minicluster-data/ts-1/wal/instance:
uuid: "1b25dd24c7a04eca99d161491588e3ba"
format_stamp: "Formatted at 2026-06-19 16:48:05 on dist-test-slave-wvz7"
server_key: "57411f61cf28c87675835c41b612c140"
server_key_iv: "a8b942b5e19d4aa744c6ec717fc42663"
server_key_version: "encryptionkey@0"
I20260619 16:48:05.772949 20447 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260619 16:48:05.776427 20461 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:48:05.777957 20447 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 16:48:05.778126 20447 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781887385903558-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781887385903558-388-0/minicluster-data/ts-1/wal
uuid: "1b25dd24c7a04eca99d161491588e3ba"
format_stamp: "Formatted at 2026-06-19 16:48:05 on dist-test-slave-wvz7"
server_key: "57411f61cf28c87675835c41b612c140"
server_key_iv: "a8b942b5e19d4aa744c6ec717fc42663"
server_key_version: "encryptionkey@0"
I20260619 16:48:05.778344 20447 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781887385903558-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781887385903558-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781887385903558-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:48:05.799877 20447 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:48:05.800762 20447 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:48:05.801057 20447 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:48:05.802142 20447 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:48:05.803727 20447 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:48:05.803830 20447 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:48:05.803941 20447 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:48:05.804016 20447 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:48:05.835790 20447 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:34637
I20260619 16:48:05.835850 20573 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:34637 every 8 connection(s)
I20260619 16:48:05.837188 20447 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781887385903558-388-0/minicluster-data/ts-1/data/info.pb
I20260619 16:48:05.844528 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu as pid 20447
I20260619 16:48:05.844722 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781887385903558-388-0/minicluster-data/ts-1/wal/instance
I20260619 16:48:05.845188 388 external_mini_cluster.cc:1428] Setting key 7d6b354be502e25c5fa9766b9c38eb6a
I20260619 16:48:05.848258 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu
/tmp/dist-test-taskEs2EH8/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781887385903558-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781887385903558-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781887385903558-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781887385903558-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35245
--builtin_ntp_servers=127.0.97.20:44289
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260619 16:48:05.850648 20445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:48:05.839713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:49689 (local address 127.0.97.62:35245)
0619 16:48:05.839896 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0619 16:48:05.839901 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:48:05.841382 (+ 1481us) server_negotiation.cc:407] Connection header received
0619 16:48:05.843277 (+ 1895us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:48:05.843284 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:48:05.843371 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:48:05.843472 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:48:05.844851 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:48:05.845634 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:48:05.847749 (+ 2115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:48:05.848131 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:48:05.848270 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:48:05.849939 (+ 1669us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:48:05.849944 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:48:05.849947 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:48:05.849966 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:48:05.849997 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:48:05.850010 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:48:05.850086 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0619 16:48:05.850341 (+ 255us) server_negotiation.cc:299] Negotiation successful
0619 16:48:05.850422 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260619 16:48:05.852035 20576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:48:05.840191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35245 (local address 127.0.97.2:49689)
0619 16:48:05.841177 (+ 986us) negotiation.cc:107] Waiting for socket to connect
0619 16:48:05.841226 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0619 16:48:05.842980 (+ 1754us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:48:05.843742 (+ 762us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:48:05.843751 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:48:05.843844 (+ 93us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:48:05.844580 (+ 736us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:48:05.844620 (+ 40us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:48:05.845841 (+ 1221us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:48:05.845846 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:48:05.847572 (+ 1726us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:48:05.847590 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:48:05.848410 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:48:05.848419 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:48:05.848748 (+ 329us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:48:05.849512 (+ 764us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 16:48:05.849517 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:48:05.849573 (+ 56us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 16:48:05.849699 (+ 126us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:48:05.850111 (+ 412us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:48:05.850122 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:48:05.850131 (+ 9us) client_negotiation.cc:769] Sending connection context
0619 16:48:05.850312 (+ 181us) client_negotiation.cc:240] Negotiation successful
0619 16:48:05.850743 (+ 431us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":533,"thread_start_us":335,"threads_started":1}
I20260619 16:48:05.853461 20574 heartbeater.cc:347] Connected to a master server at 127.0.97.62:35245
I20260619 16:48:05.853763 20574 heartbeater.cc:464] Registering TS with master...
I20260619 16:48:05.854485 20574 heartbeater.cc:511] Master 127.0.97.62:35245 requested a full tablet report, sending...
I20260619 16:48:05.855813 20257 ts_manager.cc:194] Registered new tserver with Master: 1b25dd24c7a04eca99d161491588e3ba (127.0.97.2:34637)
I20260619 16:48:05.856628 20257 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.2:49689
W20260619 16:48:06.166854 20577 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:48:06.167146 20577 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:48:06.167258 20577 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:48:06.174438 20577 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:48:06.174549 20577 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:48:06.174700 20577 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260619 16:48:06.183986 20577 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44289
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781887385903558-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781887385903558-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781887385903558-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35245
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781887385903558-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:48:06.185232 20577 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:48:06.186968 20577 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:48:06.199880 20585 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:48:06.199937 20583 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:48:06.199937 20582 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:48:06.200945 20577 server_base.cc:1034] running on GCE node
I20260619 16:48:06.201591 20577 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:48:06.202559 20577 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:48:06.203773 20577 hybrid_clock.cc:648] HybridClock initialized: now 1781887686203715 us; error 45 us; skew 500 ppm
I20260619 16:48:06.204105 20577 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 16:48:06.206384 20577 webserver.cc:460] Webserver started at http://127.0.97.3:33801/ using document root <none> and password file <none>
I20260619 16:48:06.207077 20577 fs_manager.cc:359] Metadata directory not provided
I20260619 16:48:06.207166 20577 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:48:06.207468 20577 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:48:06.210270 20577 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781887385903558-388-0/minicluster-data/ts-2/data/instance:
uuid: "1f756e0d194445ca80588faf0c7a02c3"
format_stamp: "Formatted at 2026-06-19 16:48:06 on dist-test-slave-wvz7"
server_key: "ab76df6ccf75b810fdf47a48eff39820"
server_key_iv: "ec5906305ef8dfae544a6fc17421e375"
server_key_version: "encryptionkey@0"
I20260619 16:48:06.211037 20577 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781887385903558-388-0/minicluster-data/ts-2/wal/instance:
uuid: "1f756e0d194445ca80588faf0c7a02c3"
format_stamp: "Formatted at 2026-06-19 16:48:06 on dist-test-slave-wvz7"
server_key: "ab76df6ccf75b810fdf47a48eff39820"
server_key_iv: "ec5906305ef8dfae544a6fc17421e375"
server_key_version: "encryptionkey@0"
I20260619 16:48:06.215971 20577 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 16:48:06.219235 20591 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:48:06.220515 20577 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 16:48:06.220654 20577 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781887385903558-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781887385903558-388-0/minicluster-data/ts-2/wal
uuid: "1f756e0d194445ca80588faf0c7a02c3"
format_stamp: "Formatted at 2026-06-19 16:48:06 on dist-test-slave-wvz7"
server_key: "ab76df6ccf75b810fdf47a48eff39820"
server_key_iv: "ec5906305ef8dfae544a6fc17421e375"
server_key_version: "encryptionkey@0"
I20260619 16:48:06.220862 20577 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781887385903558-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781887385903558-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781887385903558-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:48:06.257661 20577 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:48:06.258551 20577 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:48:06.258798 20577 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:48:06.259852 20577 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:48:06.262378 20577 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:48:06.262487 20577 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:48:06.262598 20577 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:48:06.262671 20577 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:48:06.295609 20577 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:42263
I20260619 16:48:06.295678 20703 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:42263 every 8 connection(s)
I20260619 16:48:06.296867 20577 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781887385903558-388-0/minicluster-data/ts-2/data/info.pb
I20260619 16:48:06.304692 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu as pid 20577
I20260619 16:48:06.304869 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781887385903558-388-0/minicluster-data/ts-2/wal/instance
I20260619 16:48:06.305421 388 external_mini_cluster.cc:1428] Setting key 815cf546e55f923ad7de5062c5d9b20a
I20260619 16:48:06.309237 20445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:48:06.299374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:35257 (local address 127.0.97.62:35245)
0619 16:48:06.299599 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0619 16:48:06.299605 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:48:06.300982 (+ 1377us) server_negotiation.cc:407] Connection header received
0619 16:48:06.302779 (+ 1797us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:48:06.302783 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:48:06.302854 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:48:06.302940 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:48:06.303961 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:48:06.304680 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:48:06.306718 (+ 2038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:48:06.307186 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:48:06.307269 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:48:06.308637 (+ 1368us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:48:06.308642 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:48:06.308645 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:48:06.308661 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:48:06.308691 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:48:06.308703 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:48:06.308771 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0619 16:48:06.308938 (+ 167us) server_negotiation.cc:299] Negotiation successful
0619 16:48:06.309011 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260619 16:48:06.310220 20706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:48:06.299949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35245 (local address 127.0.97.3:35257)
0619 16:48:06.300832 (+ 883us) negotiation.cc:107] Waiting for socket to connect
0619 16:48:06.300876 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0619 16:48:06.302530 (+ 1654us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:48:06.303162 (+ 632us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:48:06.303171 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:48:06.303262 (+ 91us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:48:06.303818 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:48:06.303835 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:48:06.304873 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:48:06.304878 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:48:06.306552 (+ 1674us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:48:06.306568 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:48:06.307504 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:48:06.307509 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:48:06.307706 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:48:06.308359 (+ 653us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 16:48:06.308362 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:48:06.308401 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 16:48:06.308482 (+ 81us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:48:06.308779 (+ 297us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:48:06.308786 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:48:06.308792 (+ 6us) client_negotiation.cc:769] Sending connection context
0619 16:48:06.308920 (+ 128us) client_negotiation.cc:240] Negotiation successful
0619 16:48:06.309230 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":427,"thread_start_us":261,"threads_started":1}
I20260619 16:48:06.311478 20704 heartbeater.cc:347] Connected to a master server at 127.0.97.62:35245
I20260619 16:48:06.311733 20704 heartbeater.cc:464] Registering TS with master...
I20260619 16:48:06.312418 20704 heartbeater.cc:511] Master 127.0.97.62:35245 requested a full tablet report, sending...
I20260619 16:48:06.313778 20257 ts_manager.cc:194] Registered new tserver with Master: 1f756e0d194445ca80588faf0c7a02c3 (127.0.97.3:42263)
I20260619 16:48:06.314545 20257 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.3:35257
I20260619 16:48:06.319998 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 16:48:06.329430 20445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:48:06.323970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34722 (local address 127.0.97.62:35245)
0619 16:48:06.324201 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0619 16:48:06.324206 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:48:06.324489 (+ 283us) server_negotiation.cc:407] Connection header received
0619 16:48:06.324611 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:48:06.324615 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:48:06.324683 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:48:06.324777 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:48:06.325356 (+ 579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:48:06.326107 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:48:06.327366 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:48:06.327741 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:48:06.327834 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:48:06.328835 (+ 1001us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:48:06.328840 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:48:06.328843 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:48:06.328860 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:48:06.328888 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:48:06.328900 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:48:06.328963 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0619 16:48:06.329078 (+ 115us) server_negotiation.cc:299] Negotiation successful
0619 16:48:06.329151 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260619 16:48:06.332855 20257 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34722:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 16:48:06.339569 20257 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 16:48:06.361156 20716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:48:06.351678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:43589 (local address 127.0.0.1:60562)
0619 16:48:06.353380 (+ 1702us) negotiation.cc:107] Waiting for socket to connect
0619 16:48:06.353427 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0619 16:48:06.353679 (+ 252us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:48:06.354593 (+ 914us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:48:06.354600 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:48:06.354630 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:48:06.355033 (+ 403us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:48:06.355046 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:48:06.356484 (+ 1438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:48:06.356488 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:48:06.358131 (+ 1643us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:48:06.358145 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:48:06.360271 (+ 2126us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:48:06.360277 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:48:06.360479 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:48:06.360528 (+ 49us) client_negotiation.cc:769] Sending connection context
0619 16:48:06.360648 (+ 120us) client_negotiation.cc:240] Negotiation successful
0619 16:48:06.360823 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1459,"thread_start_us":1316,"threads_started":1}
I20260619 16:48:06.361215 20718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:48:06.351865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60562 (local address 127.0.97.1:43589)
0619 16:48:06.352863 (+ 998us) server_negotiation.cc:206] Beginning negotiation
0619 16:48:06.352872 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 16:48:06.353534 (+ 662us) server_negotiation.cc:407] Connection header received
0619 16:48:06.353932 (+ 398us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:48:06.353940 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:48:06.354148 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:48:06.354351 (+ 203us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:48:06.355249 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:48:06.356341 (+ 1092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:48:06.358742 (+ 2401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:48:06.360091 (+ 1349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:48:06.360228 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:48:06.360475 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0619 16:48:06.360702 (+ 227us) server_negotiation.cc:299] Negotiation successful
0619 16:48:06.360877 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":824,"thread_start_us":825,"threads_started":1}
I20260619 16:48:06.365180 20378 tablet_service.cc:1467] Processing CreateTablet for tablet b07ae9da72b14b6da52728f8b46afc45 (DEFAULT_TABLE table=test-table [id=47732f4be7574cf3831666424a96419c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:48:06.365917 20717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:48:06.351832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:34637 (local address 127.0.0.1:48492)
0619 16:48:06.354127 (+ 2295us) negotiation.cc:107] Waiting for socket to connect
0619 16:48:06.354143 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 16:48:06.354266 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:48:06.359003 (+ 4737us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:48:06.359006 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:48:06.359029 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:48:06.359313 (+ 284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:48:06.359322 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:48:06.360979 (+ 1657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:48:06.360985 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:48:06.363430 (+ 2445us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:48:06.363442 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:48:06.365121 (+ 1679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:48:06.365128 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:48:06.365298 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:48:06.365330 (+ 32us) client_negotiation.cc:769] Sending connection context
0619 16:48:06.365424 (+ 94us) client_negotiation.cc:240] Negotiation successful
0619 16:48:06.365545 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2159,"mutex_wait_us":194,"thread_start_us":2029,"threads_started":1}
I20260619 16:48:06.365983 20719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:48:06.352384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48492 (local address 127.0.97.2:34637)
0619 16:48:06.353199 (+ 815us) server_negotiation.cc:206] Beginning negotiation
0619 16:48:06.353207 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 16:48:06.358110 (+ 4903us) server_negotiation.cc:407] Connection header received
0619 16:48:06.358339 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:48:06.358347 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:48:06.358605 (+ 258us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:48:06.358800 (+ 195us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:48:06.359465 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:48:06.360838 (+ 1373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:48:06.363604 (+ 2766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:48:06.364937 (+ 1333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:48:06.365068 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:48:06.365308 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0619 16:48:06.365472 (+ 164us) server_negotiation.cc:299] Negotiation successful
0619 16:48:06.365638 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":662,"thread_start_us":557,"threads_started":1}
I20260619 16:48:06.366685 20720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:48:06.354219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:42263 (local address 127.0.0.1:40168)
0619 16:48:06.358485 (+ 4266us) negotiation.cc:107] Waiting for socket to connect
0619 16:48:06.358500 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 16:48:06.358602 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:48:06.359479 (+ 877us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:48:06.359483 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:48:06.359507 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:48:06.359823 (+ 316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:48:06.359835 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:48:06.361587 (+ 1752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:48:06.361592 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:48:06.363700 (+ 2108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:48:06.363714 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:48:06.365698 (+ 1984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:48:06.365704 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:48:06.366118 (+ 414us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:48:06.366150 (+ 32us) client_negotiation.cc:769] Sending connection context
0619 16:48:06.366250 (+ 100us) client_negotiation.cc:240] Negotiation successful
0619 16:48:06.366372 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3796,"mutex_wait_us":29,"thread_start_us":3664,"threads_started":1}
I20260619 16:48:06.366915 20378 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b07ae9da72b14b6da52728f8b46afc45. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:48:06.367479 20721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:48:06.354538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40168 (local address 127.0.97.3:42263)
0619 16:48:06.358973 (+ 4435us) server_negotiation.cc:206] Beginning negotiation
0619 16:48:06.358981 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 16:48:06.359011 (+ 30us) server_negotiation.cc:407] Connection header received
0619 16:48:06.359093 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:48:06.359100 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:48:06.359325 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:48:06.359498 (+ 173us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:48:06.360008 (+ 510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:48:06.361430 (+ 1422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:48:06.363889 (+ 2459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:48:06.365217 (+ 1328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:48:06.365341 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:48:06.365578 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0619 16:48:06.366960 (+ 1382us) server_negotiation.cc:299] Negotiation successful
0619 16:48:06.367094 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4272,"thread_start_us":3777,"threads_started":1}
I20260619 16:48:06.369658 20509 tablet_service.cc:1467] Processing CreateTablet for tablet b07ae9da72b14b6da52728f8b46afc45 (DEFAULT_TABLE table=test-table [id=47732f4be7574cf3831666424a96419c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:48:06.370352 20639 tablet_service.cc:1467] Processing CreateTablet for tablet b07ae9da72b14b6da52728f8b46afc45 (DEFAULT_TABLE table=test-table [id=47732f4be7574cf3831666424a96419c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:48:06.371040 20509 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b07ae9da72b14b6da52728f8b46afc45. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:48:06.371357 20639 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b07ae9da72b14b6da52728f8b46afc45. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:48:06.382102 20722 tablet_bootstrap.cc:492] T b07ae9da72b14b6da52728f8b46afc45 P 336a948d93e44f3eb1fe2c395da95049: Bootstrap starting.
I20260619 16:48:06.382905 20723 tablet_bootstrap.cc:492] T b07ae9da72b14b6da52728f8b46afc45 P 1f756e0d194445ca80588faf0c7a02c3: Bootstrap starting.
I20260619 16:48:06.384533 20724 tablet_bootstrap.cc:492] T b07ae9da72b14b6da52728f8b46afc45 P 1b25dd24c7a04eca99d161491588e3ba: Bootstrap starting.
I20260619 16:48:06.386072 20722 tablet_bootstrap.cc:654] T b07ae9da72b14b6da52728f8b46afc45 P 336a948d93e44f3eb1fe2c395da95049: Neither blocks nor log segments found. Creating new log.
I20260619 16:48:06.387036 20722 log.cc:824] T b07ae9da72b14b6da52728f8b46afc45 P 336a948d93e44f3eb1fe2c395da95049: Log is configured to *not* fsync() on all Append() calls
I20260619 16:48:06.387315 20724 tablet_bootstrap.cc:654] T b07ae9da72b14b6da52728f8b46afc45 P 1b25dd24c7a04eca99d161491588e3ba: Neither blocks nor log segments found. Creating new log.
I20260619 16:48:06.388491 20724 log.cc:824] T b07ae9da72b14b6da52728f8b46afc45 P 1b25dd24c7a04eca99d161491588e3ba: Log is configured to *not* fsync() on all Append() calls
I20260619 16:48:06.390506 20723 tablet_bootstrap.cc:654] T b07ae9da72b14b6da52728f8b46afc45 P 1f756e0d194445ca80588faf0c7a02c3: Neither blocks nor log segments found. Creating new log.
I20260619 16:48:06.391584 20724 tablet_bootstrap.cc:492] T b07ae9da72b14b6da52728f8b46afc45 P 1b25dd24c7a04eca99d161491588e3ba: No bootstrap required, opened a new log
I20260619 16:48:06.391661 20723 log.cc:824] T b07ae9da72b14b6da52728f8b46afc45 P 1f756e0d194445ca80588faf0c7a02c3: Log is configured to *not* fsync() on all Append() calls
I20260619 16:48:06.391824 20722 tablet_bootstrap.cc:492] T b07ae9da72b14b6da52728f8b46afc45 P 336a948d93e44f3eb1fe2c395da95049: No bootstrap required, opened a new log
I20260619 16:48:06.391852 20724 ts_tablet_manager.cc:1397] T b07ae9da72b14b6da52728f8b46afc45 P 1b25dd24c7a04eca99d161491588e3ba: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.005s
I20260619 16:48:06.392077 20722 ts_tablet_manager.cc:1397] T b07ae9da72b14b6da52728f8b46afc45 P 336a948d93e44f3eb1fe2c395da95049: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.000s
I20260619 16:48:06.394605 20723 tablet_bootstrap.cc:492] T b07ae9da72b14b6da52728f8b46afc45 P 1f756e0d194445ca80588faf0c7a02c3: No bootstrap required, opened a new log
I20260619 16:48:06.394861 20723 ts_tablet_manager.cc:1397] T b07ae9da72b14b6da52728f8b46afc45 P 1f756e0d194445ca80588faf0c7a02c3: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.004s
I20260619 16:48:06.399809 20724 raft_consensus.cc:348] T b07ae9da72b14b6da52728f8b46afc45 P 1b25dd24c7a04eca99d161491588e3ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "336a948d93e44f3eb1fe2c395da95049" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43589 } } peers { permanent_uuid: "1b25dd24c7a04eca99d161491588e3ba" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34637 } } peers { permanent_uuid: "1f756e0d194445ca80588faf0c7a02c3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42263 } }
I20260619 16:48:06.400244 20724 raft_consensus.cc:374] T b07ae9da72b14b6da52728f8b46afc45 P 1b25dd24c7a04eca99d161491588e3ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:48:06.400336 20724 raft_consensus.cc:729] T b07ae9da72b14b6da52728f8b46afc45 P 1b25dd24c7a04eca99d161491588e3ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b25dd24c7a04eca99d161491588e3ba, State: Initialized, Role: FOLLOWER
I20260619 16:48:06.401088 20724 consensus_queue.cc:260] T b07ae9da72b14b6da52728f8b46afc45 P 1b25dd24c7a04eca99d161491588e3ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "336a948d93e44f3eb1fe2c395da95049" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43589 } } peers { permanent_uuid: "1b25dd24c7a04eca99d161491588e3ba" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34637 } } peers { permanent_uuid: "1f756e0d194445ca80588faf0c7a02c3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42263 } }
I20260619 16:48:06.404656 20723 raft_consensus.cc:348] T b07ae9da72b14b6da52728f8b46afc45 P 1f756e0d194445ca80588faf0c7a02c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "336a948d93e44f3eb1fe2c395da95049" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43589 } } peers { permanent_uuid: "1b25dd24c7a04eca99d161491588e3ba" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34637 } } peers { permanent_uuid: "1f756e0d194445ca80588faf0c7a02c3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42263 } }
I20260619 16:48:06.404663 20722 raft_consensus.cc:348] T b07ae9da72b14b6da52728f8b46afc45 P 336a948d93e44f3eb1fe2c395da95049 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "336a948d93e44f3eb1fe2c395da95049" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43589 } } peers { permanent_uuid: "1b25dd24c7a04eca99d161491588e3ba" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34637 } } peers { permanent_uuid: "1f756e0d194445ca80588faf0c7a02c3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42263 } }
I20260619 16:48:06.405097 20723 raft_consensus.cc:374] T b07ae9da72b14b6da52728f8b46afc45 P 1f756e0d194445ca80588faf0c7a02c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:48:06.405097 20722 raft_consensus.cc:374] T b07ae9da72b14b6da52728f8b46afc45 P 336a948d93e44f3eb1fe2c395da95049 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:48:06.405196 20722 raft_consensus.cc:729] T b07ae9da72b14b6da52728f8b46afc45 P 336a948d93e44f3eb1fe2c395da95049 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 336a948d93e44f3eb1fe2c395da95049, State: Initialized, Role: FOLLOWER
I20260619 16:48:06.405196 20723 raft_consensus.cc:729] T b07ae9da72b14b6da52728f8b46afc45 P 1f756e0d194445ca80588faf0c7a02c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f756e0d194445ca80588faf0c7a02c3, State: Initialized, Role: FOLLOWER
I20260619 16:48:06.405892 20723 consensus_queue.cc:260] T b07ae9da72b14b6da52728f8b46afc45 P 1f756e0d194445ca80588faf0c7a02c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "336a948d93e44f3eb1fe2c395da95049" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43589 } } peers { permanent_uuid: "1b25dd24c7a04eca99d161491588e3ba" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34637 } } peers { permanent_uuid: "1f756e0d194445ca80588faf0c7a02c3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42263 } }
I20260619 16:48:06.405900 20722 consensus_queue.cc:260] T b07ae9da72b14b6da52728f8b46afc45 P 336a948d93e44f3eb1fe2c395da95049 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "336a948d93e44f3eb1fe2c395da95049" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43589 } } peers { permanent_uuid: "1b25dd24c7a04eca99d161491588e3ba" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34637 } } peers { permanent_uuid: "1f756e0d194445ca80588faf0c7a02c3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42263 } }
I20260619 16:48:06.408921 20574 heartbeater.cc:503] Master 127.0.97.62:35245 was elected leader, sending a full tablet report...
I20260619 16:48:06.408957 20724 ts_tablet_manager.cc:1428] T b07ae9da72b14b6da52728f8b46afc45 P 1b25dd24c7a04eca99d161491588e3ba: Time spent starting tablet: real 0.017s user 0.010s sys 0.007s
I20260619 16:48:06.409662 20443 heartbeater.cc:503] Master 127.0.97.62:35245 was elected leader, sending a full tablet report...
I20260619 16:48:06.410308 20722 ts_tablet_manager.cc:1428] T b07ae9da72b14b6da52728f8b46afc45 P 336a948d93e44f3eb1fe2c395da95049: Time spent starting tablet: real 0.018s user 0.013s sys 0.005s
I20260619 16:48:06.411890 20704 heartbeater.cc:503] Master 127.0.97.62:35245 was elected leader, sending a full tablet report...
I20260619 16:48:06.412477 20723 ts_tablet_manager.cc:1428] T b07ae9da72b14b6da52728f8b46afc45 P 1f756e0d194445ca80588faf0c7a02c3: Time spent starting tablet: real 0.018s user 0.014s sys 0.001s
W20260619 16:48:06.548542 20705 tablet.cc:2369] T b07ae9da72b14b6da52728f8b46afc45 P 1f756e0d194445ca80588faf0c7a02c3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 16:48:06.570168 20729 raft_consensus.cc:482] T b07ae9da72b14b6da52728f8b46afc45 P 1f756e0d194445ca80588faf0c7a02c3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 16:48:06.570443 20729 raft_consensus.cc:504] T b07ae9da72b14b6da52728f8b46afc45 P 1f756e0d194445ca80588faf0c7a02c3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "336a948d93e44f3eb1fe2c395da95049" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43589 } } peers { permanent_uuid: "1b25dd24c7a04eca99d161491588e3ba" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34637 } } peers { permanent_uuid: "1f756e0d194445ca80588faf0c7a02c3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42263 } }
I20260619 16:48:06.572053 20729 leader_election.cc:288] T b07ae9da72b14b6da52728f8b46afc45 P 1f756e0d194445ca80588faf0c7a02c3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 336a948d93e44f3eb1fe2c395da95049 (127.0.97.1:43589), 1b25dd24c7a04eca99d161491588e3ba (127.0.97.2:34637)
I20260619 16:48:06.578259 20719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:48:06.572568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:46191 (local address 127.0.97.2:34637)
0619 16:48:06.572817 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0619 16:48:06.572822 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:48:06.572844 (+ 22us) server_negotiation.cc:407] Connection header received
0619 16:48:06.572916 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:48:06.572922 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:48:06.573005 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:48:06.573094 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:48:06.573563 (+ 469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:48:06.574512 (+ 949us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:48:06.576470 (+ 1958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:48:06.577468 (+ 998us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:48:06.577560 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:48:06.577633 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0619 16:48:06.577968 (+ 335us) server_negotiation.cc:299] Negotiation successful
0619 16:48:06.578059 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260619 16:48:06.578346 20706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:48:06.572402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:34637 (local address 127.0.97.3:46191)
0619 16:48:06.572665 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0619 16:48:06.572685 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 16:48:06.572811 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:48:06.573191 (+ 380us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:48:06.573196 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:48:06.573238 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:48:06.573390 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:48:06.573400 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:48:06.574655 (+ 1255us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:48:06.574661 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:48:06.576313 (+ 1652us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:48:06.576325 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:48:06.577612 (+ 1287us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:48:06.577618 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:48:06.577814 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:48:06.577849 (+ 35us) client_negotiation.cc:769] Sending connection context
0619 16:48:06.577941 (+ 92us) client_negotiation.cc:240] Negotiation successful
0619 16:48:06.578075 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146}
I20260619 16:48:06.579016 20731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:48:06.572438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:43589 (local address 127.0.97.3:45691)
0619 16:48:06.573342 (+ 904us) negotiation.cc:107] Waiting for socket to connect
0619 16:48:06.573358 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 16:48:06.573452 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:48:06.573751 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:48:06.573756 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:48:06.573781 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:48:06.574130 (+ 349us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:48:06.574141 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:48:06.575455 (+ 1314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:48:06.575459 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:48:06.576997 (+ 1538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:48:06.577011 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:48:06.578382 (+ 1371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:48:06.578388 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:48:06.578550 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:48:06.578583 (+ 33us) client_negotiation.cc:769] Sending connection context
0619 16:48:06.578709 (+ 126us) client_negotiation.cc:240] Negotiation successful
0619 16:48:06.578804 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":807,"thread_start_us":578,"threads_started":1}
I20260619 16:48:06.579140 20718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:48:06.572555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:45691 (local address 127.0.97.1:43589)
0619 16:48:06.572873 (+ 318us) server_negotiation.cc:206] Beginning negotiation
0619 16:48:06.572880 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:48:06.573446 (+ 566us) server_negotiation.cc:407] Connection header received
0619 16:48:06.573559 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:48:06.573563 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:48:06.573629 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:48:06.573695 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:48:06.574327 (+ 632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:48:06.575303 (+ 976us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:48:06.577181 (+ 1878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:48:06.578194 (+ 1013us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:48:06.578311 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:48:06.578393 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0619 16:48:06.578777 (+ 384us) server_negotiation.cc:299] Negotiation successful
0619 16:48:06.578901 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99}
I20260619 16:48:06.579509 20529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b07ae9da72b14b6da52728f8b46afc45" candidate_uuid: "1f756e0d194445ca80588faf0c7a02c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b25dd24c7a04eca99d161491588e3ba" is_pre_election: true
I20260619 16:48:06.580063 20529 raft_consensus.cc:2454] T b07ae9da72b14b6da52728f8b46afc45 P 1b25dd24c7a04eca99d161491588e3ba [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f756e0d194445ca80588faf0c7a02c3 in term 0.
I20260619 16:48:06.580093 20398 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b07ae9da72b14b6da52728f8b46afc45" candidate_uuid: "1f756e0d194445ca80588faf0c7a02c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "336a948d93e44f3eb1fe2c395da95049" is_pre_election: true
I20260619 16:48:06.580591 20398 raft_consensus.cc:2454] T b07ae9da72b14b6da52728f8b46afc45 P 336a948d93e44f3eb1fe2c395da95049 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f756e0d194445ca80588faf0c7a02c3 in term 0.
I20260619 16:48:06.580693 20595 leader_election.cc:302] T b07ae9da72b14b6da52728f8b46afc45 P 1f756e0d194445ca80588faf0c7a02c3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b25dd24c7a04eca99d161491588e3ba, 1f756e0d194445ca80588faf0c7a02c3; no voters:
I20260619 16:48:06.581151 20729 raft_consensus.cc:2781] T b07ae9da72b14b6da52728f8b46afc45 P 1f756e0d194445ca80588faf0c7a02c3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 16:48:06.581275 20729 raft_consensus.cc:482] T b07ae9da72b14b6da52728f8b46afc45 P 1f756e0d194445ca80588faf0c7a02c3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 16:48:06.581386 20729 raft_consensus.cc:3037] T b07ae9da72b14b6da52728f8b46afc45 P 1f756e0d194445ca80588faf0c7a02c3 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:48:06.583621 20729 raft_consensus.cc:504] T b07ae9da72b14b6da52728f8b46afc45 P 1f756e0d194445ca80588faf0c7a02c3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "336a948d93e44f3eb1fe2c395da95049" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43589 } } peers { permanent_uuid: "1b25dd24c7a04eca99d161491588e3ba" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34637 } } peers { permanent_uuid: "1f756e0d194445ca80588faf0c7a02c3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42263 } }
I20260619 16:48:06.584208 20729 leader_election.cc:288] T b07ae9da72b14b6da52728f8b46afc45 P 1f756e0d194445ca80588faf0c7a02c3 [CANDIDATE]: Term 1 election: Requested vote from peers 336a948d93e44f3eb1fe2c395da95049 (127.0.97.1:43589), 1b25dd24c7a04eca99d161491588e3ba (127.0.97.2:34637)
I20260619 16:48:06.584688 20398 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b07ae9da72b14b6da52728f8b46afc45" candidate_uuid: "1f756e0d194445ca80588faf0c7a02c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "336a948d93e44f3eb1fe2c395da95049"
I20260619 16:48:06.584919 20398 raft_consensus.cc:3037] T b07ae9da72b14b6da52728f8b46afc45 P 336a948d93e44f3eb1fe2c395da95049 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:48:06.585822 20529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b07ae9da72b14b6da52728f8b46afc45" candidate_uuid: "1f756e0d194445ca80588faf0c7a02c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b25dd24c7a04eca99d161491588e3ba"
I20260619 16:48:06.586030 20529 raft_consensus.cc:3037] T b07ae9da72b14b6da52728f8b46afc45 P 1b25dd24c7a04eca99d161491588e3ba [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:48:06.587587 20398 raft_consensus.cc:2454] T b07ae9da72b14b6da52728f8b46afc45 P 336a948d93e44f3eb1fe2c395da95049 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f756e0d194445ca80588faf0c7a02c3 in term 1.
I20260619 16:48:06.587924 20593 leader_election.cc:302] T b07ae9da72b14b6da52728f8b46afc45 P 1f756e0d194445ca80588faf0c7a02c3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f756e0d194445ca80588faf0c7a02c3, 336a948d93e44f3eb1fe2c395da95049; no voters:
I20260619 16:48:06.588196 20729 raft_consensus.cc:2781] T b07ae9da72b14b6da52728f8b46afc45 P 1f756e0d194445ca80588faf0c7a02c3 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:48:06.588653 20729 raft_consensus.cc:686] T b07ae9da72b14b6da52728f8b46afc45 P 1f756e0d194445ca80588faf0c7a02c3 [term 1 LEADER]: Becoming Leader. State: Replica: 1f756e0d194445ca80588faf0c7a02c3, State: Running, Role: LEADER
I20260619 16:48:06.588716 20529 raft_consensus.cc:2454] T b07ae9da72b14b6da52728f8b46afc45 P 1b25dd24c7a04eca99d161491588e3ba [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f756e0d194445ca80588faf0c7a02c3 in term 1.
W20260619 16:48:06.589108 20575 tablet.cc:2369] T b07ae9da72b14b6da52728f8b46afc45 P 1b25dd24c7a04eca99d161491588e3ba: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 16:48:06.589212 20729 consensus_queue.cc:237] T b07ae9da72b14b6da52728f8b46afc45 P 1f756e0d194445ca80588faf0c7a02c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "336a948d93e44f3eb1fe2c395da95049" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43589 } } peers { permanent_uuid: "1b25dd24c7a04eca99d161491588e3ba" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34637 } } peers { permanent_uuid: "1f756e0d194445ca80588faf0c7a02c3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42263 } }
I20260619 16:48:06.593645 20255 catalog_manager.cc:5495] T b07ae9da72b14b6da52728f8b46afc45 P 1f756e0d194445ca80588faf0c7a02c3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1f756e0d194445ca80588faf0c7a02c3 (127.0.97.3). New cstate: current_term: 1 leader_uuid: "1f756e0d194445ca80588faf0c7a02c3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "336a948d93e44f3eb1fe2c395da95049" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43589 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1b25dd24c7a04eca99d161491588e3ba" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34637 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1f756e0d194445ca80588faf0c7a02c3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42263 } health_report { overall_health: HEALTHY } } }
W20260619 16:48:06.650810 20444 tablet.cc:2369] T b07ae9da72b14b6da52728f8b46afc45 P 336a948d93e44f3eb1fe2c395da95049: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 16:48:06.655675 20721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:48:06.650642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40182 (local address 127.0.97.3:42263)
0619 16:48:06.650841 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0619 16:48:06.650845 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:48:06.650859 (+ 14us) server_negotiation.cc:407] Connection header received
0619 16:48:06.650950 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:48:06.650954 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:48:06.651053 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:48:06.651186 (+ 133us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 16:48:06.651535 (+ 349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:48:06.652262 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:48:06.653622 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:48:06.654044 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:48:06.654152 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:48:06.654617 (+ 465us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 16:48:06.655128 (+ 511us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 16:48:06.655259 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0619 16:48:06.655370 (+ 111us) server_negotiation.cc:299] Negotiation successful
0619 16:48:06.655446 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260619 16:48:06.663807 20398 raft_consensus.cc:1261] T b07ae9da72b14b6da52728f8b46afc45 P 336a948d93e44f3eb1fe2c395da95049 [term 1 FOLLOWER]: Refusing update from remote peer 1f756e0d194445ca80588faf0c7a02c3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:48:06.663830 20529 raft_consensus.cc:1261] T b07ae9da72b14b6da52728f8b46afc45 P 1b25dd24c7a04eca99d161491588e3ba [term 1 FOLLOWER]: Refusing update from remote peer 1f756e0d194445ca80588faf0c7a02c3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:48:06.664860 20732 consensus_queue.cc:1035] T b07ae9da72b14b6da52728f8b46afc45 P 1f756e0d194445ca80588faf0c7a02c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b25dd24c7a04eca99d161491588e3ba" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34637 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:48:06.665220 20729 consensus_queue.cc:1035] T b07ae9da72b14b6da52728f8b46afc45 P 1f756e0d194445ca80588faf0c7a02c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "336a948d93e44f3eb1fe2c395da95049" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43589 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:48:06.681347 20737 mvcc.cc:205] Tried to move back new op lower bound from 7298611964564140032 to 7298611964273561600. Current Snapshot: MvccSnapshot[applied={T|T < 7298611964564140032}]
I20260619 16:48:06.692294 20256 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34722:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:48:06.692613 20256 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34722:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:48:06.696651 20256 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 15a9096772d043a9b21a26c0971240ac: Sending DeleteTablet for 3 replicas of tablet b07ae9da72b14b6da52728f8b46afc45
I20260619 16:48:06.698028 20378 tablet_service.cc:1514] Processing DeleteTablet for tablet b07ae9da72b14b6da52728f8b46afc45 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:48:06 UTC) from {username='slave'} at 127.0.0.1:60562
I20260619 16:48:06.698036 20509 tablet_service.cc:1514] Processing DeleteTablet for tablet b07ae9da72b14b6da52728f8b46afc45 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:48:06 UTC) from {username='slave'} at 127.0.0.1:48492
I20260619 16:48:06.698932 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu pid 20316
I20260619 16:48:06.698976 20639 tablet_service.cc:1514] Processing DeleteTablet for tablet b07ae9da72b14b6da52728f8b46afc45 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:48:06 UTC) from {username='slave'} at 127.0.0.1:40168
I20260619 16:48:06.706284 20746 tablet_replica.cc:330] T b07ae9da72b14b6da52728f8b46afc45 P 336a948d93e44f3eb1fe2c395da95049: stopping tablet replica
I20260619 16:48:06.706750 20746 raft_consensus.cc:2229] T b07ae9da72b14b6da52728f8b46afc45 P 336a948d93e44f3eb1fe2c395da95049 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 16:48:06.707996 20746 raft_consensus.cc:2258] T b07ae9da72b14b6da52728f8b46afc45 P 336a948d93e44f3eb1fe2c395da95049 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:48:06.708069 20745 tablet_replica.cc:330] T b07ae9da72b14b6da52728f8b46afc45 P 1b25dd24c7a04eca99d161491588e3ba: stopping tablet replica
I20260619 16:48:06.708455 20745 raft_consensus.cc:2229] T b07ae9da72b14b6da52728f8b46afc45 P 1b25dd24c7a04eca99d161491588e3ba [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 16:48:06.709002 20745 raft_consensus.cc:2258] T b07ae9da72b14b6da52728f8b46afc45 P 1b25dd24c7a04eca99d161491588e3ba [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:48:06.711136 20746 ts_tablet_manager.cc:1905] T b07ae9da72b14b6da52728f8b46afc45 P 336a948d93e44f3eb1fe2c395da95049: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:48:06.711440 20745 ts_tablet_manager.cc:1905] T b07ae9da72b14b6da52728f8b46afc45 P 1b25dd24c7a04eca99d161491588e3ba: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:48:06.713320 20747 tablet_replica.cc:330] T b07ae9da72b14b6da52728f8b46afc45 P 1f756e0d194445ca80588faf0c7a02c3: stopping tablet replica
I20260619 16:48:06.714107 20747 raft_consensus.cc:2229] T b07ae9da72b14b6da52728f8b46afc45 P 1f756e0d194445ca80588faf0c7a02c3 [term 1 LEADER]: Raft consensus shutting down.
I20260619 16:48:06.715068 20747 raft_consensus.cc:2258] T b07ae9da72b14b6da52728f8b46afc45 P 1f756e0d194445ca80588faf0c7a02c3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:48:06.718031 20745 ts_tablet_manager.cc:1918] T b07ae9da72b14b6da52728f8b46afc45 P 1b25dd24c7a04eca99d161491588e3ba: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 16:48:06.718156 20746 ts_tablet_manager.cc:1918] T b07ae9da72b14b6da52728f8b46afc45 P 336a948d93e44f3eb1fe2c395da95049: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 16:48:06.718187 20745 log.cc:1196] T b07ae9da72b14b6da52728f8b46afc45 P 1b25dd24c7a04eca99d161491588e3ba: Deleting WAL directory at /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781887385903558-388-0/minicluster-data/ts-1/wal/wals/b07ae9da72b14b6da52728f8b46afc45
I20260619 16:48:06.718321 20746 log.cc:1196] T b07ae9da72b14b6da52728f8b46afc45 P 336a948d93e44f3eb1fe2c395da95049: Deleting WAL directory at /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781887385903558-388-0/minicluster-data/ts-0/wal/wals/b07ae9da72b14b6da52728f8b46afc45
I20260619 16:48:06.718724 20745 ts_tablet_manager.cc:1939] T b07ae9da72b14b6da52728f8b46afc45 P 1b25dd24c7a04eca99d161491588e3ba: Deleting consensus metadata
I20260619 16:48:06.718851 20746 ts_tablet_manager.cc:1939] T b07ae9da72b14b6da52728f8b46afc45 P 336a948d93e44f3eb1fe2c395da95049: Deleting consensus metadata
I20260619 16:48:06.720345 20718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:48:06.700348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60576 (local address 127.0.97.1:43589)
0619 16:48:06.700560 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0619 16:48:06.700568 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 16:48:06.708922 (+ 8354us) server_negotiation.cc:407] Connection header received
0619 16:48:06.709119 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:48:06.709124 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:48:06.709227 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:48:06.709319 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:48:06.710073 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:48:06.711152 (+ 1079us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:48:06.715302 (+ 4150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:48:06.715947 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:48:06.716039 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:48:06.718300 (+ 2261us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:48:06.718314 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:48:06.718330 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:48:06.719122 (+ 792us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:48:06.719180 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:48:06.719205 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:48:06.719302 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0619 16:48:06.719892 (+ 590us) server_negotiation.cc:299] Negotiation successful
0619 16:48:06.719998 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260619 16:48:06.720624 20244 catalog_manager.cc:4841] TS 1b25dd24c7a04eca99d161491588e3ba (127.0.97.2:34637): tablet b07ae9da72b14b6da52728f8b46afc45 (table test-table [id=47732f4be7574cf3831666424a96419c]) successfully deleted
I20260619 16:48:06.721067 20242 catalog_manager.cc:4841] TS 336a948d93e44f3eb1fe2c395da95049 (127.0.97.1:43589): tablet b07ae9da72b14b6da52728f8b46afc45 (table test-table [id=47732f4be7574cf3831666424a96419c]) successfully deleted
I20260619 16:48:06.721225 20747 ts_tablet_manager.cc:1905] T b07ae9da72b14b6da52728f8b46afc45 P 1f756e0d194445ca80588faf0c7a02c3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:48:06.721576 20438 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:48:06.728636 20747 ts_tablet_manager.cc:1918] T b07ae9da72b14b6da52728f8b46afc45 P 1f756e0d194445ca80588faf0c7a02c3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 16:48:06.728832 20747 log.cc:1196] T b07ae9da72b14b6da52728f8b46afc45 P 1f756e0d194445ca80588faf0c7a02c3: Deleting WAL directory at /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781887385903558-388-0/minicluster-data/ts-2/wal/wals/b07ae9da72b14b6da52728f8b46afc45
I20260619 16:48:06.729307 20747 ts_tablet_manager.cc:1939] T b07ae9da72b14b6da52728f8b46afc45 P 1f756e0d194445ca80588faf0c7a02c3: Deleting consensus metadata
I20260619 16:48:06.731011 20244 catalog_manager.cc:4841] TS 1f756e0d194445ca80588faf0c7a02c3 (127.0.97.3:42263): tablet b07ae9da72b14b6da52728f8b46afc45 (table test-table [id=47732f4be7574cf3831666424a96419c]) successfully deleted
I20260619 16:48:06.812266 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu with pid 20316
I20260619 16:48:06.826761 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu pid 20447
I20260619 16:48:06.833478 20719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:48:06.827513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48496 (local address 127.0.97.2:34637)
0619 16:48:06.827690 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0619 16:48:06.827695 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:48:06.827714 (+ 19us) server_negotiation.cc:407] Connection header received
0619 16:48:06.827906 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:48:06.827911 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:48:06.828008 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:48:06.828092 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:48:06.828514 (+ 422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:48:06.829265 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:48:06.830644 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:48:06.831043 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:48:06.831151 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:48:06.832196 (+ 1045us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:48:06.832205 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:48:06.832219 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:48:06.832782 (+ 563us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:48:06.832827 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:48:06.832847 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:48:06.832947 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0619 16:48:06.833073 (+ 126us) server_negotiation.cc:299] Negotiation successful
0619 16:48:06.833176 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260619 16:48:06.834226 20569 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:48:06.924281 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu with pid 20447
I20260619 16:48:06.938382 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu pid 20577
I20260619 16:48:06.945324 20721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:48:06.939109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40198 (local address 127.0.97.3:42263)
0619 16:48:06.939356 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0619 16:48:06.939362 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:48:06.939382 (+ 20us) server_negotiation.cc:407] Connection header received
0619 16:48:06.939554 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:48:06.939560 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:48:06.939636 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:48:06.939728 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:48:06.940136 (+ 408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:48:06.940927 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:48:06.942274 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:48:06.942699 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:48:06.942807 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:48:06.943940 (+ 1133us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:48:06.943949 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:48:06.943973 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:48:06.944540 (+ 567us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:48:06.944587 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:48:06.944608 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:48:06.944714 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0619 16:48:06.944858 (+ 144us) server_negotiation.cc:299] Negotiation successful
0619 16:48:06.944961 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125}
I20260619 16:48:06.946061 20699 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:48:07.063423 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu with pid 20577
I20260619 16:48:07.078018 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu pid 20226
I20260619 16:48:07.079012 20287 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:48:07.186885 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu with pid 20226
2026-06-19T16:48:07Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (2807 ms)
[----------] 6 tests from EncryptionPolicyTest (19936 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781887385903558-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 16:48:07 dist-test-slave-wvz7 krb5kdc[20755](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 16:48:07 dist-test-slave-wvz7 krb5kdc[20755](info): set up 2 sockets
Jun 19 16:48:07 dist-test-slave-wvz7 krb5kdc[20755](info): commencing operation
krb5kdc: starting...
W20260619 16:48:08.211202 388 mini_kdc.cc:121] Time spent starting KDC: real 0.986s user 0.000s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781887385903558-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781887385903558-388-0/krb5kdc/test-user.keytab.
Jun 19 16:48:08 dist-test-slave-wvz7 krb5kdc[20755](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887688, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T16:48:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T16:48:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781887385903558-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781887385903558-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 16:48:08.450927 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu
/tmp/dist-test-taskEs2EH8/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781887385903558-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781887385903558-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781887385903558-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781887385903558-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45911
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781887385903558-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46565
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45911
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781887385903558-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781887385903558-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781887385903558-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:48:08.763919 20768 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:48:08.764200 20768 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:48:08.764319 20768 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:48:08.771639 20768 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:48:08.771759 20768 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:48:08.771816 20768 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:48:08.771898 20768 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:48:08.771953 20768 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:48:08.781754 20768 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46565
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781887385903558-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781887385903558-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45911
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781887385903558-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45911
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781887385903558-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781887385903558-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:48:08.782986 20768 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:48:08.784258 20768 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:48:08.795452 20773 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:48:08.795536 20774 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:48:08.795595 20776 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:48:08.796660 20768 server_base.cc:1034] running on GCE node
I20260619 16:48:08.797492 20768 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:48:08.798944 20768 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:48:08.800170 20768 hybrid_clock.cc:648] HybridClock initialized: now 1781887688800130 us; error 37 us; skew 500 ppm
I20260619 16:48:08.800503 20768 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:48:08 dist-test-slave-wvz7 krb5kdc[20755](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887688, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:48:08.805410 20768 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 16:48:08.806901 20768 webserver.cc:460] Webserver started at http://127.0.0.1:46223/ using document root <none> and password file <none>
I20260619 16:48:08.807547 20768 fs_manager.cc:359] Metadata directory not provided
I20260619 16:48:08.807655 20768 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:48:08.807948 20768 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:48:08.810856 20768 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781887385903558-388-0/minicluster-data/master-0/data/instance:
uuid: "9cd31de887f146d19c1850036b336312"
format_stamp: "Formatted at 2026-06-19 16:48:08 on dist-test-slave-wvz7"
server_key: "46a12e383954b83d59768e7116aab073"
server_key_iv: "6b34a83ed172bb3c0d43cea2b833a033"
server_key_version: "encryptionkey@0"
I20260619 16:48:08.811601 20768 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781887385903558-388-0/minicluster-data/master-0/wal/instance:
uuid: "9cd31de887f146d19c1850036b336312"
format_stamp: "Formatted at 2026-06-19 16:48:08 on dist-test-slave-wvz7"
server_key: "46a12e383954b83d59768e7116aab073"
server_key_iv: "6b34a83ed172bb3c0d43cea2b833a033"
server_key_version: "encryptionkey@0"
I20260619 16:48:08.816707 20768 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260619 16:48:08.820187 20783 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:48:08.821556 20768 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260619 16:48:08.821719 20768 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781887385903558-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781887385903558-388-0/minicluster-data/master-0/wal
uuid: "9cd31de887f146d19c1850036b336312"
format_stamp: "Formatted at 2026-06-19 16:48:08 on dist-test-slave-wvz7"
server_key: "46a12e383954b83d59768e7116aab073"
server_key_iv: "6b34a83ed172bb3c0d43cea2b833a033"
server_key_version: "encryptionkey@0"
I20260619 16:48:08.821959 20768 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781887385903558-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781887385903558-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781887385903558-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:48:08.835590 20768 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:48:08.839576 20768 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:48:08.839835 20768 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:48:08.857029 20768 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45911
I20260619 16:48:08.857057 20834 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45911 every 8 connection(s)
I20260619 16:48:08.858361 20768 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781887385903558-388-0/minicluster-data/master-0/data/info.pb
I20260619 16:48:08.862044 20835 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:48:08.863283 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu as pid 20768
I20260619 16:48:08.863474 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781887385903558-388-0/minicluster-data/master-0/wal/instance
I20260619 16:48:08.864066 388 external_mini_cluster.cc:1428] Setting key 6c8b0412137e9217735ca45b3c809a59
I20260619 16:48:08.875779 20835 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9cd31de887f146d19c1850036b336312: Bootstrap starting.
Jun 19 16:48:08 dist-test-slave-wvz7 krb5kdc[20755](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887688, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260619 16:48:08.879755 20835 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9cd31de887f146d19c1850036b336312: Neither blocks nor log segments found. Creating new log.
I20260619 16:48:08.880918 20835 log.cc:824] T 00000000000000000000000000000000 P 9cd31de887f146d19c1850036b336312: Log is configured to *not* fsync() on all Append() calls
I20260619 16:48:08.884068 20835 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9cd31de887f146d19c1850036b336312: No bootstrap required, opened a new log
I20260619 16:48:08.888069 20838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:48:08.865912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56041 (local address 127.0.0.1:45911)
0619 16:48:08.866665 (+ 753us) server_negotiation.cc:206] Beginning negotiation
0619 16:48:08.866676 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 16:48:08.866718 (+ 42us) server_negotiation.cc:407] Connection header received
0619 16:48:08.868317 (+ 1599us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:48:08.868342 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:48:08.868751 (+ 409us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:48:08.869071 (+ 320us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:48:08.870959 (+ 1888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:48:08.872259 (+ 1300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:48:08.874065 (+ 1806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:48:08.874765 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:48:08.874908 (+ 143us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:48:08.879851 (+ 4943us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:48:08.879859 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:48:08.879877 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:48:08.879929 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:48:08.883090 (+ 3161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:48:08.884092 (+ 1002us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:48:08.884101 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:48:08.884109 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:48:08.884355 (+ 246us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:48:08.884878 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:48:08.884883 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:48:08.884886 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:48:08.885583 (+ 697us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:48:08.885929 (+ 346us) server_negotiation.cc:1035] Waiting for connection context
0619 16:48:08.886360 (+ 431us) server_negotiation.cc:299] Negotiation successful
0619 16:48:08.886714 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":543,"thread_start_us":347,"threads_started":1}
I20260619 16:48:08.891868 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 16:48:08.892910 20835 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9cd31de887f146d19c1850036b336312 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9cd31de887f146d19c1850036b336312" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45911 } }
I20260619 16:48:08.893231 20835 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9cd31de887f146d19c1850036b336312 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:48:08.893326 20835 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9cd31de887f146d19c1850036b336312 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9cd31de887f146d19c1850036b336312, State: Initialized, Role: FOLLOWER
I20260619 16:48:08.893884 20835 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9cd31de887f146d19c1850036b336312 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9cd31de887f146d19c1850036b336312" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45911 } }
I20260619 16:48:08.894095 20835 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9cd31de887f146d19c1850036b336312 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:48:08.894217 20835 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9cd31de887f146d19c1850036b336312 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:48:08.894356 20835 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9cd31de887f146d19c1850036b336312 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:48:08.896675 20835 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9cd31de887f146d19c1850036b336312 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9cd31de887f146d19c1850036b336312" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45911 } }
I20260619 16:48:08.897224 20835 leader_election.cc:302] T 00000000000000000000000000000000 P 9cd31de887f146d19c1850036b336312 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9cd31de887f146d19c1850036b336312; no voters:
I20260619 16:48:08.897871 20835 leader_election.cc:288] T 00000000000000000000000000000000 P 9cd31de887f146d19c1850036b336312 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 16:48:08.897971 20845 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9cd31de887f146d19c1850036b336312 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:48:08.899080 20845 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9cd31de887f146d19c1850036b336312 [term 1 LEADER]: Becoming Leader. State: Replica: 9cd31de887f146d19c1850036b336312, State: Running, Role: LEADER
I20260619 16:48:08.899708 20845 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9cd31de887f146d19c1850036b336312 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9cd31de887f146d19c1850036b336312" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45911 } }
I20260619 16:48:08.900103 20835 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9cd31de887f146d19c1850036b336312 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:48:08.903123 20847 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9cd31de887f146d19c1850036b336312 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9cd31de887f146d19c1850036b336312. Latest consensus state: current_term: 1 leader_uuid: "9cd31de887f146d19c1850036b336312" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9cd31de887f146d19c1850036b336312" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45911 } } }
I20260619 16:48:08.903820 20847 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9cd31de887f146d19c1850036b336312 [sys.catalog]: This master's current role is: LEADER
I20260619 16:48:08.902508 20846 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9cd31de887f146d19c1850036b336312 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9cd31de887f146d19c1850036b336312" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9cd31de887f146d19c1850036b336312" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45911 } } }
I20260619 16:48:08.904527 20846 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9cd31de887f146d19c1850036b336312 [sys.catalog]: This master's current role is: LEADER
I20260619 16:48:08.911206 20851 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:48:08.913350 20838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:48:08.896407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60745 (local address 127.0.0.1:45911)
0619 16:48:08.896585 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0619 16:48:08.896591 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:48:08.896925 (+ 334us) server_negotiation.cc:407] Connection header received
0619 16:48:08.897079 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:48:08.897085 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:48:08.897159 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:48:08.897280 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:48:08.898856 (+ 1576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:48:08.899779 (+ 923us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:48:08.901126 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:48:08.901798 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:48:08.901909 (+ 111us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:48:08.905125 (+ 3216us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:48:08.905132 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:48:08.905138 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:48:08.905181 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:48:08.908924 (+ 3743us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:48:08.909986 (+ 1062us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:48:08.909995 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:48:08.909998 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:48:08.910179 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:48:08.910790 (+ 611us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:48:08.910796 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:48:08.910798 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:48:08.911262 (+ 464us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:48:08.911541 (+ 279us) server_negotiation.cc:1035] Waiting for connection context
0619 16:48:08.912686 (+ 1145us) server_negotiation.cc:299] Negotiation successful
0619 16:48:08.912889 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260619 16:48:08.916227 20851 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:48:08.924332 20851 catalog_manager.cc:1347] Generated new cluster ID: 88ed085eaec041768f7abf4fdaa7855a
I20260619 16:48:08.924445 20851 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:48:08.941314 20851 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 16:48:08.942726 20851 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:48:08.954380 20851 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9cd31de887f146d19c1850036b336312: Generated new TSK 0
I20260619 16:48:08.955166 20851 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 16:48:08.966349 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu pid 20768
I20260619 16:48:08.967434 20830 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:48:09.059442 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu with pid 20768
W20260619 16:48:09.073500 20753 connection.cc:535] client connection to 127.0.0.1:45911 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:45911): Connection reset by peer (error 104)
2026-06-19T16:48:09Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1871 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781887385903558-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 16:48:09 dist-test-slave-wvz7 krb5kdc[20867](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 16:48:09 dist-test-slave-wvz7 krb5kdc[20867](info): set up 2 sockets
Jun 19 16:48:09 dist-test-slave-wvz7 krb5kdc[20867](info): commencing operation
krb5kdc: starting...
W20260619 16:48:10.074342 388 mini_kdc.cc:121] Time spent starting KDC: real 0.978s user 0.005s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781887385903558-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781887385903558-388-0/krb5kdc/test-user.keytab.
Jun 19 16:48:10 dist-test-slave-wvz7 krb5kdc[20867](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887690, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T16:48:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T16:48:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781887385903558-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781887385903558-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 16:48:10.312749 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu
/tmp/dist-test-taskEs2EH8/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781887385903558-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781887385903558-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781887385903558-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781887385903558-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43241
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781887385903558-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:32925
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43241
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781887385903558-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781887385903558-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781887385903558-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:48:10.632018 20880 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:48:10.632308 20880 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:48:10.632396 20880 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:48:10.639561 20880 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:48:10.639674 20880 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:48:10.639722 20880 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:48:10.639781 20880 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:48:10.639824 20880 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:48:10.649547 20880 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:32925
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781887385903558-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781887385903558-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43241
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781887385903558-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43241
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781887385903558-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781887385903558-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:48:10.650786 20880 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:48:10.652005 20880 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:48:10.663084 20888 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:48:10.663051 20886 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:48:10.663129 20885 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:48:10.663848 20880 server_base.cc:1034] running on GCE node
I20260619 16:48:10.664697 20880 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:48:10.666230 20880 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:48:10.667443 20880 hybrid_clock.cc:648] HybridClock initialized: now 1781887690667390 us; error 42 us; skew 500 ppm
I20260619 16:48:10.667791 20880 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:48:10 dist-test-slave-wvz7 krb5kdc[20867](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887690, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:48:10.670892 20880 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 16:48:10.672261 20880 webserver.cc:460] Webserver started at http://127.0.0.1:35343/ using document root <none> and password file <none>
I20260619 16:48:10.672890 20880 fs_manager.cc:359] Metadata directory not provided
I20260619 16:48:10.673002 20880 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:48:10.673282 20880 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:48:10.676222 20880 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781887385903558-388-0/minicluster-data/master-0/data/instance:
uuid: "1d1077c25e10495d9ec9f4cb885ec59f"
format_stamp: "Formatted at 2026-06-19 16:48:10 on dist-test-slave-wvz7"
server_key: "76453384c99e963a780df778e47ca854"
server_key_iv: "af1d19a1fc49b4b63f8dc8884e5828d5"
server_key_version: "encryptionkey@0"
I20260619 16:48:10.676913 20880 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781887385903558-388-0/minicluster-data/master-0/wal/instance:
uuid: "1d1077c25e10495d9ec9f4cb885ec59f"
format_stamp: "Formatted at 2026-06-19 16:48:10 on dist-test-slave-wvz7"
server_key: "76453384c99e963a780df778e47ca854"
server_key_iv: "af1d19a1fc49b4b63f8dc8884e5828d5"
server_key_version: "encryptionkey@0"
I20260619 16:48:10.681839 20880 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.004s
I20260619 16:48:10.685091 20895 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:48:10.686404 20880 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260619 16:48:10.686555 20880 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781887385903558-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781887385903558-388-0/minicluster-data/master-0/wal
uuid: "1d1077c25e10495d9ec9f4cb885ec59f"
format_stamp: "Formatted at 2026-06-19 16:48:10 on dist-test-slave-wvz7"
server_key: "76453384c99e963a780df778e47ca854"
server_key_iv: "af1d19a1fc49b4b63f8dc8884e5828d5"
server_key_version: "encryptionkey@0"
I20260619 16:48:10.686721 20880 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781887385903558-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781887385903558-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781887385903558-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:48:10.704758 20880 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:48:10.708675 20880 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:48:10.708925 20880 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:48:10.725792 20880 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43241
I20260619 16:48:10.725850 20946 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43241 every 8 connection(s)
I20260619 16:48:10.727083 20880 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781887385903558-388-0/minicluster-data/master-0/data/info.pb
I20260619 16:48:10.730712 20947 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:48:10.735713 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu as pid 20880
I20260619 16:48:10.735885 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781887385903558-388-0/minicluster-data/master-0/wal/instance
I20260619 16:48:10.736377 388 external_mini_cluster.cc:1428] Setting key 5c6f19aee3b4bc105227dd52ce56827e
I20260619 16:48:10.741387 20947 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1d1077c25e10495d9ec9f4cb885ec59f: Bootstrap starting.
I20260619 16:48:10.744848 20947 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1d1077c25e10495d9ec9f4cb885ec59f: Neither blocks nor log segments found. Creating new log.
I20260619 16:48:10.745954 20947 log.cc:824] T 00000000000000000000000000000000 P 1d1077c25e10495d9ec9f4cb885ec59f: Log is configured to *not* fsync() on all Append() calls
I20260619 16:48:10.748797 20947 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1d1077c25e10495d9ec9f4cb885ec59f: No bootstrap required, opened a new log
I20260619 16:48:10.749241 20950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:48:10.738104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58777 (local address 127.0.0.1:43241)
0619 16:48:10.738907 (+ 803us) server_negotiation.cc:206] Beginning negotiation
0619 16:48:10.738919 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 16:48:10.738963 (+ 44us) server_negotiation.cc:407] Connection header received
0619 16:48:10.740621 (+ 1658us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:48:10.740643 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:48:10.741030 (+ 387us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:48:10.741320 (+ 290us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:48:10.741879 (+ 559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:48:10.743198 (+ 1319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:48:10.745004 (+ 1806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:48:10.745687 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:48:10.745860 (+ 173us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:48:10.747100 (+ 1240us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:48:10.747111 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:48:10.747129 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:48:10.747655 (+ 526us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:48:10.747722 (+ 67us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:48:10.747745 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:48:10.747856 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0619 16:48:10.748130 (+ 274us) server_negotiation.cc:299] Negotiation successful
0619 16:48:10.748288 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":581,"thread_start_us":384,"threads_started":1}
I20260619 16:48:10.752892 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 16:48:10.757768 20947 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1d1077c25e10495d9ec9f4cb885ec59f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d1077c25e10495d9ec9f4cb885ec59f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43241 } }
I20260619 16:48:10.758224 20947 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1d1077c25e10495d9ec9f4cb885ec59f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:48:10.758334 20947 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1d1077c25e10495d9ec9f4cb885ec59f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d1077c25e10495d9ec9f4cb885ec59f, State: Initialized, Role: FOLLOWER
I20260619 16:48:10.758956 20947 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1d1077c25e10495d9ec9f4cb885ec59f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d1077c25e10495d9ec9f4cb885ec59f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43241 } }
I20260619 16:48:10.759239 20947 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1d1077c25e10495d9ec9f4cb885ec59f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:48:10.759382 20947 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1d1077c25e10495d9ec9f4cb885ec59f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:48:10.759596 20947 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1d1077c25e10495d9ec9f4cb885ec59f [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:48:10.762502 20947 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1d1077c25e10495d9ec9f4cb885ec59f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d1077c25e10495d9ec9f4cb885ec59f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43241 } }
I20260619 16:48:10.763008 20947 leader_election.cc:302] T 00000000000000000000000000000000 P 1d1077c25e10495d9ec9f4cb885ec59f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1d1077c25e10495d9ec9f4cb885ec59f; no voters:
I20260619 16:48:10.763607 20947 leader_election.cc:288] T 00000000000000000000000000000000 P 1d1077c25e10495d9ec9f4cb885ec59f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 16:48:10.763705 20957 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1d1077c25e10495d9ec9f4cb885ec59f [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:48:10.764341 20957 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1d1077c25e10495d9ec9f4cb885ec59f [term 1 LEADER]: Becoming Leader. State: Replica: 1d1077c25e10495d9ec9f4cb885ec59f, State: Running, Role: LEADER
I20260619 16:48:10.764957 20957 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1d1077c25e10495d9ec9f4cb885ec59f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d1077c25e10495d9ec9f4cb885ec59f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43241 } }
I20260619 16:48:10.765893 20947 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1d1077c25e10495d9ec9f4cb885ec59f [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:48:10.768883 20959 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d1077c25e10495d9ec9f4cb885ec59f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1d1077c25e10495d9ec9f4cb885ec59f. Latest consensus state: current_term: 1 leader_uuid: "1d1077c25e10495d9ec9f4cb885ec59f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d1077c25e10495d9ec9f4cb885ec59f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43241 } } }
I20260619 16:48:10.769265 20959 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d1077c25e10495d9ec9f4cb885ec59f [sys.catalog]: This master's current role is: LEADER
I20260619 16:48:10.769651 20950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:48:10.757965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47079 (local address 127.0.0.1:43241)
0619 16:48:10.758117 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0619 16:48:10.758122 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:48:10.758438 (+ 316us) server_negotiation.cc:407] Connection header received
0619 16:48:10.758617 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:48:10.758621 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:48:10.758700 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:48:10.758791 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:48:10.759458 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:48:10.760290 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:48:10.762183 (+ 1893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:48:10.762642 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:48:10.762758 (+ 116us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:48:10.768458 (+ 5700us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:48:10.768464 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:48:10.768468 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:48:10.768492 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:48:10.768571 (+ 79us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:48:10.768594 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:48:10.768702 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0619 16:48:10.769208 (+ 506us) server_negotiation.cc:299] Negotiation successful
0619 16:48:10.769288 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260619 16:48:10.770519 20958 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d1077c25e10495d9ec9f4cb885ec59f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1d1077c25e10495d9ec9f4cb885ec59f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d1077c25e10495d9ec9f4cb885ec59f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43241 } } }
I20260619 16:48:10.770831 20958 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d1077c25e10495d9ec9f4cb885ec59f [sys.catalog]: This master's current role is: LEADER
I20260619 16:48:10.772630 20963 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:48:10.778461 20963 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:48:10.787058 20963 catalog_manager.cc:1347] Generated new cluster ID: 27127cf9e00c4de0be931eb205ff0c59
I20260619 16:48:10.787221 20963 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:48:10.799384 20963 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 16:48:10.800693 20963 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:48:10.811648 20963 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1d1077c25e10495d9ec9f4cb885ec59f: Generated new TSK 0
I20260619 16:48:10.812444 20963 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 16:48:10.820950 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu pid 20880
W20260619 16:48:10.822167 20942 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:58777
W20260619 16:48:10.822971 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu pid 20880: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 16:48:10.823170 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu with pid 20880
W20260619 16:48:10.836200 20865 connection.cc:535] client connection to 127.0.0.1:43241 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:43241): Connection reset by peer (error 104)
2026-06-19T16:48:10Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1758 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781887385903558-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 16:48:10 dist-test-slave-wvz7 krb5kdc[20978](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 16:48:10 dist-test-slave-wvz7 krb5kdc[20978](info): set up 2 sockets
Jun 19 16:48:10 dist-test-slave-wvz7 krb5kdc[20978](info): commencing operation
krb5kdc: starting...
W20260619 16:48:11.829650 388 mini_kdc.cc:121] Time spent starting KDC: real 0.975s user 0.000s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781887385903558-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781887385903558-388-0/krb5kdc/test-user.keytab.
Jun 19 16:48:11 dist-test-slave-wvz7 krb5kdc[20978](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887691, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T16:48:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T16:48:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781887385903558-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781887385903558-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 16:48:12.072749 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu
/tmp/dist-test-taskEs2EH8/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781887385903558-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781887385903558-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781887385903558-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781887385903558-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42835
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781887385903558-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34303
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42835
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781887385903558-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781887385903558-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781887385903558-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:48:12.399641 20991 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:48:12.399930 20991 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:48:12.400029 20991 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:48:12.407217 20991 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:48:12.407330 20991 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:48:12.407374 20991 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:48:12.407434 20991 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:48:12.407476 20991 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:48:12.417309 20991 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34303
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781887385903558-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781887385903558-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42835
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781887385903558-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42835
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781887385903558-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781887385903558-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:48:12.418567 20991 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:48:12.419791 20991 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:48:12.434144 20997 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:48:12.434170 20999 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:48:12.434557 20996 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:48:12.435056 20991 server_base.cc:1034] running on GCE node
I20260619 16:48:12.435804 20991 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:48:12.437252 20991 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:48:12.438459 20991 hybrid_clock.cc:648] HybridClock initialized: now 1781887692438415 us; error 43 us; skew 500 ppm
I20260619 16:48:12.438808 20991 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:48:12 dist-test-slave-wvz7 krb5kdc[20978](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887692, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:48:12.441879 20991 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 16:48:12.443352 20991 webserver.cc:460] Webserver started at http://127.0.0.1:39177/ using document root <none> and password file <none>
I20260619 16:48:12.444006 20991 fs_manager.cc:359] Metadata directory not provided
I20260619 16:48:12.444126 20991 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:48:12.444414 20991 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:48:12.447433 20991 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781887385903558-388-0/minicluster-data/master-0/data/instance:
uuid: "d4e016f24f70448dad5c0bf326d2af77"
format_stamp: "Formatted at 2026-06-19 16:48:12 on dist-test-slave-wvz7"
server_key: "b2bd5be02602edeb137bf0ebc786594e"
server_key_iv: "7068d9637b6b316df3ac0554950dae10"
server_key_version: "encryptionkey@0"
I20260619 16:48:12.448283 20991 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781887385903558-388-0/minicluster-data/master-0/wal/instance:
uuid: "d4e016f24f70448dad5c0bf326d2af77"
format_stamp: "Formatted at 2026-06-19 16:48:12 on dist-test-slave-wvz7"
server_key: "b2bd5be02602edeb137bf0ebc786594e"
server_key_iv: "7068d9637b6b316df3ac0554950dae10"
server_key_version: "encryptionkey@0"
I20260619 16:48:12.453456 20991 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260619 16:48:12.457008 21006 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:48:12.458432 20991 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 16:48:12.458598 20991 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781887385903558-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781887385903558-388-0/minicluster-data/master-0/wal
uuid: "d4e016f24f70448dad5c0bf326d2af77"
format_stamp: "Formatted at 2026-06-19 16:48:12 on dist-test-slave-wvz7"
server_key: "b2bd5be02602edeb137bf0ebc786594e"
server_key_iv: "7068d9637b6b316df3ac0554950dae10"
server_key_version: "encryptionkey@0"
I20260619 16:48:12.458791 20991 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781887385903558-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781887385903558-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781887385903558-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:48:12.464641 20991 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:48:12.468768 20991 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:48:12.469043 20991 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:48:12.485981 20991 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42835
I20260619 16:48:12.486045 21057 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42835 every 8 connection(s)
I20260619 16:48:12.487339 20991 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781887385903558-388-0/minicluster-data/master-0/data/info.pb
I20260619 16:48:12.490896 21058 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:48:12.495512 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu as pid 20991
I20260619 16:48:12.495687 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781887385903558-388-0/minicluster-data/master-0/wal/instance
I20260619 16:48:12.496230 388 external_mini_cluster.cc:1428] Setting key 989771ca0c28c7c13951dac1edac7364
I20260619 16:48:12.502003 21058 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d4e016f24f70448dad5c0bf326d2af77: Bootstrap starting.
I20260619 16:48:12.503939 21061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:48:12.498011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34739 (local address 127.0.0.1:42835)
0619 16:48:12.498781 (+ 770us) server_negotiation.cc:206] Beginning negotiation
0619 16:48:12.498797 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0619 16:48:12.498839 (+ 42us) server_negotiation.cc:407] Connection header received
0619 16:48:12.500582 (+ 1743us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:48:12.500605 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:48:12.500963 (+ 358us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:48:12.501282 (+ 319us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:48:12.502367 (+ 1085us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:48:12.502374 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:48:12.502387 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:48:12.502403 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:48:12.502468 (+ 65us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:48:12.502492 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:48:12.502665 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0619 16:48:12.502773 (+ 108us) server_negotiation.cc:299] Negotiation successful
0619 16:48:12.502945 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":556,"thread_start_us":348,"threads_started":1}
I20260619 16:48:12.505200 21058 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d4e016f24f70448dad5c0bf326d2af77: Neither blocks nor log segments found. Creating new log.
I20260619 16:48:12.506443 21058 log.cc:824] T 00000000000000000000000000000000 P d4e016f24f70448dad5c0bf326d2af77: Log is configured to *not* fsync() on all Append() calls
I20260619 16:48:12.507673 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 16:48:12.509029 21058 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d4e016f24f70448dad5c0bf326d2af77: No bootstrap required, opened a new log
I20260619 16:48:12.516036 21061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:48:12.511748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33851 (local address 127.0.0.1:42835)
0619 16:48:12.511891 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0619 16:48:12.511895 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:48:12.514211 (+ 2316us) server_negotiation.cc:407] Connection header received
0619 16:48:12.514371 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:48:12.514374 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:48:12.514421 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:48:12.514505 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:48:12.515402 (+ 897us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:48:12.515407 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:48:12.515412 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:48:12.515428 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:48:12.515474 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:48:12.515489 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:48:12.515592 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0619 16:48:12.515704 (+ 112us) server_negotiation.cc:299] Negotiation successful
0619 16:48:12.515763 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260619 16:48:12.517495 21058 raft_consensus.cc:348] T 00000000000000000000000000000000 P d4e016f24f70448dad5c0bf326d2af77 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4e016f24f70448dad5c0bf326d2af77" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42835 } }
I20260619 16:48:12.517882 21058 raft_consensus.cc:374] T 00000000000000000000000000000000 P d4e016f24f70448dad5c0bf326d2af77 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:48:12.517972 21058 raft_consensus.cc:729] T 00000000000000000000000000000000 P d4e016f24f70448dad5c0bf326d2af77 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4e016f24f70448dad5c0bf326d2af77, State: Initialized, Role: FOLLOWER
I20260619 16:48:12.518522 21058 consensus_queue.cc:260] T 00000000000000000000000000000000 P d4e016f24f70448dad5c0bf326d2af77 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4e016f24f70448dad5c0bf326d2af77" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42835 } }
I20260619 16:48:12.518752 21058 raft_consensus.cc:388] T 00000000000000000000000000000000 P d4e016f24f70448dad5c0bf326d2af77 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:48:12.518875 21058 raft_consensus.cc:482] T 00000000000000000000000000000000 P d4e016f24f70448dad5c0bf326d2af77 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:48:12.519026 21058 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d4e016f24f70448dad5c0bf326d2af77 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:48:12.521195 21058 raft_consensus.cc:504] T 00000000000000000000000000000000 P d4e016f24f70448dad5c0bf326d2af77 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4e016f24f70448dad5c0bf326d2af77" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42835 } }
I20260619 16:48:12.521698 21058 leader_election.cc:302] T 00000000000000000000000000000000 P d4e016f24f70448dad5c0bf326d2af77 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d4e016f24f70448dad5c0bf326d2af77; no voters:
I20260619 16:48:12.522408 21058 leader_election.cc:288] T 00000000000000000000000000000000 P d4e016f24f70448dad5c0bf326d2af77 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 16:48:12.522516 21068 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d4e016f24f70448dad5c0bf326d2af77 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:48:12.523136 21068 raft_consensus.cc:686] T 00000000000000000000000000000000 P d4e016f24f70448dad5c0bf326d2af77 [term 1 LEADER]: Becoming Leader. State: Replica: d4e016f24f70448dad5c0bf326d2af77, State: Running, Role: LEADER
I20260619 16:48:12.523705 21068 consensus_queue.cc:237] T 00000000000000000000000000000000 P d4e016f24f70448dad5c0bf326d2af77 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4e016f24f70448dad5c0bf326d2af77" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42835 } }
I20260619 16:48:12.524597 21058 sys_catalog.cc:564] T 00000000000000000000000000000000 P d4e016f24f70448dad5c0bf326d2af77 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:48:12.526691 21070 sys_catalog.cc:455] T 00000000000000000000000000000000 P d4e016f24f70448dad5c0bf326d2af77 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d4e016f24f70448dad5c0bf326d2af77. Latest consensus state: current_term: 1 leader_uuid: "d4e016f24f70448dad5c0bf326d2af77" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4e016f24f70448dad5c0bf326d2af77" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42835 } } }
I20260619 16:48:12.527119 21070 sys_catalog.cc:458] T 00000000000000000000000000000000 P d4e016f24f70448dad5c0bf326d2af77 [sys.catalog]: This master's current role is: LEADER
I20260619 16:48:12.526623 21069 sys_catalog.cc:455] T 00000000000000000000000000000000 P d4e016f24f70448dad5c0bf326d2af77 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d4e016f24f70448dad5c0bf326d2af77" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4e016f24f70448dad5c0bf326d2af77" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42835 } } }
I20260619 16:48:12.527387 21069 sys_catalog.cc:458] T 00000000000000000000000000000000 P d4e016f24f70448dad5c0bf326d2af77 [sys.catalog]: This master's current role is: LEADER
I20260619 16:48:12.528786 21074 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:48:12.533845 21074 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:48:12.542937 21074 catalog_manager.cc:1347] Generated new cluster ID: 89fe643dd63f4f88a21c30afb95a5fb4
I20260619 16:48:12.543104 21074 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:48:12.553275 21074 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 16:48:12.553709 21074 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:48:12.566902 21074 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d4e016f24f70448dad5c0bf326d2af77: Generated new TSK 0
I20260619 16:48:12.567749 21074 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 16:48:12.577991 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu pid 20991
W20260619 16:48:12.579001 21053 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:34739
W20260619 16:48:12.579658 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu pid 20991: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 16:48:12.579782 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu with pid 20991
2026-06-19T16:48:12Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1756 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781887385903558-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 16:48:12 dist-test-slave-wvz7 krb5kdc[21089](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 16:48:12 dist-test-slave-wvz7 krb5kdc[21089](info): set up 2 sockets
Jun 19 16:48:12 dist-test-slave-wvz7 krb5kdc[21089](info): commencing operation
krb5kdc: starting...
W20260619 16:48:13.593886 388 mini_kdc.cc:121] Time spent starting KDC: real 0.982s user 0.002s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781887385903558-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781887385903558-388-0/krb5kdc/test-user.keytab.
Jun 19 16:48:13 dist-test-slave-wvz7 krb5kdc[21089](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887693, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T16:48:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T16:48:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781887385903558-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781887385903558-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 16:48:13.826723 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu
/tmp/dist-test-taskEs2EH8/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781887385903558-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781887385903558-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781887385903558-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781887385903558-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38685
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781887385903558-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40839
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38685
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781887385903558-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781887385903558-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781887385903558-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:48:14.143507 21102 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:48:14.143843 21102 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:48:14.143987 21102 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:48:14.151362 21102 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:48:14.151481 21102 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:48:14.151558 21102 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:48:14.151643 21102 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:48:14.151701 21102 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:48:14.161826 21102 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40839
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781887385903558-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781887385903558-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38685
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781887385903558-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38685
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781887385903558-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781887385903558-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:48:14.163070 21102 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:48:14.164310 21102 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:48:14.175599 21108 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:48:14.175676 21110 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:48:14.175722 21107 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:48:14.176460 21102 server_base.cc:1034] running on GCE node
I20260619 16:48:14.177238 21102 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:48:14.178723 21102 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:48:14.179925 21102 hybrid_clock.cc:648] HybridClock initialized: now 1781887694179879 us; error 41 us; skew 500 ppm
I20260619 16:48:14.180260 21102 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:48:14 dist-test-slave-wvz7 krb5kdc[21089](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887694, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:48:14.183350 21102 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 16:48:14.184808 21102 webserver.cc:460] Webserver started at http://127.0.0.1:36659/ using document root <none> and password file <none>
I20260619 16:48:14.185478 21102 fs_manager.cc:359] Metadata directory not provided
I20260619 16:48:14.185603 21102 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:48:14.185926 21102 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:48:14.188874 21102 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781887385903558-388-0/minicluster-data/master-0/data/instance:
uuid: "5fff9b89d3554f23a7e7ea4d116962d5"
format_stamp: "Formatted at 2026-06-19 16:48:14 on dist-test-slave-wvz7"
server_key: "56fb093bd6dd3f1baf784e1dd9d5ef9c"
server_key_iv: "7290c0e305415570fbee58ec305df3f4"
server_key_version: "encryptionkey@0"
I20260619 16:48:14.189605 21102 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781887385903558-388-0/minicluster-data/master-0/wal/instance:
uuid: "5fff9b89d3554f23a7e7ea4d116962d5"
format_stamp: "Formatted at 2026-06-19 16:48:14 on dist-test-slave-wvz7"
server_key: "56fb093bd6dd3f1baf784e1dd9d5ef9c"
server_key_iv: "7290c0e305415570fbee58ec305df3f4"
server_key_version: "encryptionkey@0"
I20260619 16:48:14.194638 21102 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260619 16:48:14.198105 21117 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:48:14.199388 21102 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260619 16:48:14.199543 21102 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781887385903558-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781887385903558-388-0/minicluster-data/master-0/wal
uuid: "5fff9b89d3554f23a7e7ea4d116962d5"
format_stamp: "Formatted at 2026-06-19 16:48:14 on dist-test-slave-wvz7"
server_key: "56fb093bd6dd3f1baf784e1dd9d5ef9c"
server_key_iv: "7290c0e305415570fbee58ec305df3f4"
server_key_version: "encryptionkey@0"
I20260619 16:48:14.199736 21102 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781887385903558-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781887385903558-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781887385903558-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:48:14.214267 21102 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:48:14.218166 21102 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:48:14.218425 21102 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:48:14.235364 21102 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38685
I20260619 16:48:14.235383 21168 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38685 every 8 connection(s)
I20260619 16:48:14.236686 21102 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781887385903558-388-0/minicluster-data/master-0/data/info.pb
I20260619 16:48:14.238991 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu as pid 21102
I20260619 16:48:14.239181 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781887385903558-388-0/minicluster-data/master-0/wal/instance
I20260619 16:48:14.239738 388 external_mini_cluster.cc:1428] Setting key 7cd12311fcf7153185526437f3ffc5b6
I20260619 16:48:14.240867 21169 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:48:14.252311 21169 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5fff9b89d3554f23a7e7ea4d116962d5: Bootstrap starting.
Jun 19 16:48:14 dist-test-slave-wvz7 krb5kdc[21089](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887693, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260619 16:48:14.256090 21169 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5fff9b89d3554f23a7e7ea4d116962d5: Neither blocks nor log segments found. Creating new log.
I20260619 16:48:14.257423 21169 log.cc:824] T 00000000000000000000000000000000 P 5fff9b89d3554f23a7e7ea4d116962d5: Log is configured to *not* fsync() on all Append() calls
I20260619 16:48:14.260769 21169 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5fff9b89d3554f23a7e7ea4d116962d5: No bootstrap required, opened a new log
I20260619 16:48:14.262120 21172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:48:14.241498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46767 (local address 127.0.0.1:38685)
0619 16:48:14.242258 (+ 760us) server_negotiation.cc:206] Beginning negotiation
0619 16:48:14.242270 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 16:48:14.242309 (+ 39us) server_negotiation.cc:407] Connection header received
0619 16:48:14.243877 (+ 1568us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:48:14.243900 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:48:14.244313 (+ 413us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:48:14.244701 (+ 388us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:48:14.246154 (+ 1453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:48:14.247452 (+ 1298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:48:14.248827 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:48:14.249530 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:48:14.249698 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:48:14.253519 (+ 3821us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:48:14.253530 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:48:14.253548 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:48:14.253602 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:48:14.257452 (+ 3850us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:48:14.258269 (+ 817us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:48:14.258276 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:48:14.258282 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:48:14.258450 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:48:14.258884 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:48:14.258889 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:48:14.258892 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:48:14.259530 (+ 638us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:48:14.259833 (+ 303us) server_negotiation.cc:1035] Waiting for connection context
0619 16:48:14.260248 (+ 415us) server_negotiation.cc:299] Negotiation successful
0619 16:48:14.260734 (+ 486us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":556,"thread_start_us":349,"threads_started":1}
I20260619 16:48:14.266296 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 16:48:14.269337 21169 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5fff9b89d3554f23a7e7ea4d116962d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fff9b89d3554f23a7e7ea4d116962d5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38685 } }
I20260619 16:48:14.269685 21169 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5fff9b89d3554f23a7e7ea4d116962d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:48:14.269790 21169 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5fff9b89d3554f23a7e7ea4d116962d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fff9b89d3554f23a7e7ea4d116962d5, State: Initialized, Role: FOLLOWER
I20260619 16:48:14.270367 21169 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5fff9b89d3554f23a7e7ea4d116962d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fff9b89d3554f23a7e7ea4d116962d5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38685 } }
I20260619 16:48:14.270659 21169 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5fff9b89d3554f23a7e7ea4d116962d5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:48:14.270784 21169 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5fff9b89d3554f23a7e7ea4d116962d5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:48:14.270949 21169 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5fff9b89d3554f23a7e7ea4d116962d5 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:48:14.273059 21169 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5fff9b89d3554f23a7e7ea4d116962d5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fff9b89d3554f23a7e7ea4d116962d5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38685 } }
I20260619 16:48:14.273550 21169 leader_election.cc:302] T 00000000000000000000000000000000 P 5fff9b89d3554f23a7e7ea4d116962d5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5fff9b89d3554f23a7e7ea4d116962d5; no voters:
I20260619 16:48:14.274175 21169 leader_election.cc:288] T 00000000000000000000000000000000 P 5fff9b89d3554f23a7e7ea4d116962d5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 16:48:14.274274 21179 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5fff9b89d3554f23a7e7ea4d116962d5 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:48:14.276437 21169 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5fff9b89d3554f23a7e7ea4d116962d5 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:48:14.281134 21179 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5fff9b89d3554f23a7e7ea4d116962d5 [term 1 LEADER]: Becoming Leader. State: Replica: 5fff9b89d3554f23a7e7ea4d116962d5, State: Running, Role: LEADER
I20260619 16:48:14.281932 21179 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5fff9b89d3554f23a7e7ea4d116962d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fff9b89d3554f23a7e7ea4d116962d5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38685 } }
I20260619 16:48:14.284538 21172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:48:14.270361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46931 (local address 127.0.0.1:38685)
0619 16:48:14.270540 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0619 16:48:14.270545 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:48:14.270901 (+ 356us) server_negotiation.cc:407] Connection header received
0619 16:48:14.271064 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:48:14.271069 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:48:14.271146 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:48:14.271231 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:48:14.272573 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:48:14.273311 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:48:14.274669 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:48:14.275314 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:48:14.275429 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:48:14.278744 (+ 3315us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:48:14.278751 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:48:14.278755 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:48:14.278786 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:48:14.281871 (+ 3085us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:48:14.282537 (+ 666us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:48:14.282543 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:48:14.282547 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:48:14.282692 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:48:14.283074 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:48:14.283079 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:48:14.283083 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:48:14.283352 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:48:14.283645 (+ 293us) server_negotiation.cc:1035] Waiting for connection context
0619 16:48:14.283944 (+ 299us) server_negotiation.cc:299] Negotiation successful
0619 16:48:14.284148 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260619 16:48:14.291280 21180 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5fff9b89d3554f23a7e7ea4d116962d5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5fff9b89d3554f23a7e7ea4d116962d5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fff9b89d3554f23a7e7ea4d116962d5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38685 } } }
I20260619 16:48:14.292145 21180 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5fff9b89d3554f23a7e7ea4d116962d5 [sys.catalog]: This master's current role is: LEADER
I20260619 16:48:14.291558 21181 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5fff9b89d3554f23a7e7ea4d116962d5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5fff9b89d3554f23a7e7ea4d116962d5. Latest consensus state: current_term: 1 leader_uuid: "5fff9b89d3554f23a7e7ea4d116962d5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fff9b89d3554f23a7e7ea4d116962d5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38685 } } }
I20260619 16:48:14.292618 21181 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5fff9b89d3554f23a7e7ea4d116962d5 [sys.catalog]: This master's current role is: LEADER
I20260619 16:48:14.293244 21192 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:48:14.297906 21192 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:48:14.305693 21192 catalog_manager.cc:1347] Generated new cluster ID: b0713d44de6844d3bb303723e03b5503
I20260619 16:48:14.305828 21192 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:48:14.323025 21192 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 16:48:14.324856 21192 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:48:14.335692 21192 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5fff9b89d3554f23a7e7ea4d116962d5: Generated new TSK 0
I20260619 16:48:14.336705 21192 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 16:48:14.344587 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu pid 21102
I20260619 16:48:14.345623 21164 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:48:14.435041 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu with pid 21102
2026-06-19T16:48:14Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1855 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781887385903558-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 16:48:14 dist-test-slave-wvz7 krb5kdc[21201](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 16:48:14 dist-test-slave-wvz7 krb5kdc[21201](info): set up 2 sockets
Jun 19 16:48:14 dist-test-slave-wvz7 krb5kdc[21201](info): commencing operation
krb5kdc: starting...
W20260619 16:48:15.461869 388 mini_kdc.cc:121] Time spent starting KDC: real 0.995s user 0.003s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781887385903558-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781887385903558-388-0/krb5kdc/test-user.keytab.
Jun 19 16:48:15 dist-test-slave-wvz7 krb5kdc[21201](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887695, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T16:48:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T16:48:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781887385903558-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781887385903558-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 16:48:15.697171 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu
/tmp/dist-test-taskEs2EH8/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781887385903558-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781887385903558-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781887385903558-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781887385903558-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44709
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781887385903558-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34731
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44709
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781887385903558-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781887385903558-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781887385903558-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:48:16.018239 21214 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:48:16.018530 21214 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:48:16.018641 21214 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:48:16.025874 21214 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:48:16.025995 21214 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:48:16.026047 21214 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:48:16.026129 21214 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:48:16.026181 21214 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:48:16.036099 21214 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34731
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781887385903558-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781887385903558-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44709
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781887385903558-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44709
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781887385903558-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781887385903558-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:48:16.037392 21214 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:48:16.038678 21214 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:48:16.050031 21220 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:48:16.050061 21222 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:48:16.050107 21219 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:48:16.050673 21214 server_base.cc:1034] running on GCE node
I20260619 16:48:16.051430 21214 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:48:16.052821 21214 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:48:16.054072 21214 hybrid_clock.cc:648] HybridClock initialized: now 1781887696054000 us; error 62 us; skew 500 ppm
I20260619 16:48:16.054423 21214 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:48:16 dist-test-slave-wvz7 krb5kdc[21201](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887696, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:48:16.062213 21214 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 16:48:16.063746 21214 webserver.cc:460] Webserver started at http://127.0.0.1:32983/ using document root <none> and password file <none>
I20260619 16:48:16.064443 21214 fs_manager.cc:359] Metadata directory not provided
I20260619 16:48:16.064574 21214 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:48:16.064889 21214 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:48:16.068285 21214 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781887385903558-388-0/minicluster-data/master-0/data/instance:
uuid: "8f73a5e2b0e141058321e69b91aaf3f3"
format_stamp: "Formatted at 2026-06-19 16:48:16 on dist-test-slave-wvz7"
server_key: "2589ecdf3c0290c5345b8e12c89bd730"
server_key_iv: "e29b586f6e371cac14262d47797f2d8f"
server_key_version: "encryptionkey@0"
I20260619 16:48:16.069078 21214 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781887385903558-388-0/minicluster-data/master-0/wal/instance:
uuid: "8f73a5e2b0e141058321e69b91aaf3f3"
format_stamp: "Formatted at 2026-06-19 16:48:16 on dist-test-slave-wvz7"
server_key: "2589ecdf3c0290c5345b8e12c89bd730"
server_key_iv: "e29b586f6e371cac14262d47797f2d8f"
server_key_version: "encryptionkey@0"
I20260619 16:48:16.074613 21214 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260619 16:48:16.077936 21229 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:48:16.079193 21214 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260619 16:48:16.079355 21214 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781887385903558-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781887385903558-388-0/minicluster-data/master-0/wal
uuid: "8f73a5e2b0e141058321e69b91aaf3f3"
format_stamp: "Formatted at 2026-06-19 16:48:16 on dist-test-slave-wvz7"
server_key: "2589ecdf3c0290c5345b8e12c89bd730"
server_key_iv: "e29b586f6e371cac14262d47797f2d8f"
server_key_version: "encryptionkey@0"
I20260619 16:48:16.079537 21214 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781887385903558-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781887385903558-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781887385903558-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:48:16.095880 21214 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:48:16.100390 21214 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:48:16.100680 21214 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:48:16.119621 21214 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44709
I20260619 16:48:16.119640 21280 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44709 every 8 connection(s)
I20260619 16:48:16.121138 21214 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781887385903558-388-0/minicluster-data/master-0/data/info.pb
I20260619 16:48:16.124866 21281 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:48:16.130354 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu as pid 21214
I20260619 16:48:16.130551 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781887385903558-388-0/minicluster-data/master-0/wal/instance
I20260619 16:48:16.130932 388 external_mini_cluster.cc:1428] Setting key 0fa3c6f51628baef1e71a438e2b1fd1a
I20260619 16:48:16.135577 21281 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8f73a5e2b0e141058321e69b91aaf3f3: Bootstrap starting.
I20260619 16:48:16.139120 21281 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8f73a5e2b0e141058321e69b91aaf3f3: Neither blocks nor log segments found. Creating new log.
I20260619 16:48:16.140318 21281 log.cc:824] T 00000000000000000000000000000000 P 8f73a5e2b0e141058321e69b91aaf3f3: Log is configured to *not* fsync() on all Append() calls
I20260619 16:48:16.143098 21281 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8f73a5e2b0e141058321e69b91aaf3f3: No bootstrap required, opened a new log
I20260619 16:48:16.144207 21284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:48:16.132697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53805 (local address 127.0.0.1:44709)
0619 16:48:16.133579 (+ 882us) server_negotiation.cc:206] Beginning negotiation
0619 16:48:16.133591 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 16:48:16.133633 (+ 42us) server_negotiation.cc:407] Connection header received
0619 16:48:16.135434 (+ 1801us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:48:16.135456 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:48:16.135912 (+ 456us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:48:16.136247 (+ 335us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:48:16.136845 (+ 598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:48:16.138197 (+ 1352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:48:16.139995 (+ 1798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:48:16.140720 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:48:16.140890 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:48:16.142014 (+ 1124us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:48:16.142021 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:48:16.142036 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:48:16.142547 (+ 511us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:48:16.142622 (+ 75us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:48:16.142645 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:48:16.142747 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0619 16:48:16.142962 (+ 215us) server_negotiation.cc:299] Negotiation successful
0619 16:48:16.143229 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":647,"thread_start_us":431,"threads_started":1}
I20260619 16:48:16.147656 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 16:48:16.150844 21281 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8f73a5e2b0e141058321e69b91aaf3f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f73a5e2b0e141058321e69b91aaf3f3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44709 } }
I20260619 16:48:16.151198 21281 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8f73a5e2b0e141058321e69b91aaf3f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:48:16.151294 21281 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8f73a5e2b0e141058321e69b91aaf3f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f73a5e2b0e141058321e69b91aaf3f3, State: Initialized, Role: FOLLOWER
I20260619 16:48:16.151916 21281 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8f73a5e2b0e141058321e69b91aaf3f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f73a5e2b0e141058321e69b91aaf3f3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44709 } }
I20260619 16:48:16.152145 21281 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8f73a5e2b0e141058321e69b91aaf3f3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:48:16.152281 21281 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8f73a5e2b0e141058321e69b91aaf3f3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:48:16.152443 21281 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8f73a5e2b0e141058321e69b91aaf3f3 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:48:16.154870 21281 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8f73a5e2b0e141058321e69b91aaf3f3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f73a5e2b0e141058321e69b91aaf3f3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44709 } }
I20260619 16:48:16.155361 21281 leader_election.cc:302] T 00000000000000000000000000000000 P 8f73a5e2b0e141058321e69b91aaf3f3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8f73a5e2b0e141058321e69b91aaf3f3; no voters:
I20260619 16:48:16.155969 21281 leader_election.cc:288] T 00000000000000000000000000000000 P 8f73a5e2b0e141058321e69b91aaf3f3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 16:48:16.156050 21291 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8f73a5e2b0e141058321e69b91aaf3f3 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:48:16.157079 21291 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8f73a5e2b0e141058321e69b91aaf3f3 [term 1 LEADER]: Becoming Leader. State: Replica: 8f73a5e2b0e141058321e69b91aaf3f3, State: Running, Role: LEADER
I20260619 16:48:16.157763 21284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:48:16.151603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45537 (local address 127.0.0.1:44709)
0619 16:48:16.151764 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0619 16:48:16.151769 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:48:16.152124 (+ 355us) server_negotiation.cc:407] Connection header received
0619 16:48:16.152296 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:48:16.152301 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:48:16.152364 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:48:16.152456 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:48:16.152987 (+ 531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:48:16.153727 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:48:16.155039 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:48:16.155499 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:48:16.155610 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:48:16.156785 (+ 1175us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:48:16.156791 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:48:16.156796 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:48:16.156816 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:48:16.156867 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:48:16.156886 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:48:16.156969 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0619 16:48:16.157391 (+ 422us) server_negotiation.cc:299] Negotiation successful
0619 16:48:16.157475 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260619 16:48:16.157766 21291 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8f73a5e2b0e141058321e69b91aaf3f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f73a5e2b0e141058321e69b91aaf3f3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44709 } }
I20260619 16:48:16.158311 21281 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8f73a5e2b0e141058321e69b91aaf3f3 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:48:16.160651 21293 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8f73a5e2b0e141058321e69b91aaf3f3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8f73a5e2b0e141058321e69b91aaf3f3. Latest consensus state: current_term: 1 leader_uuid: "8f73a5e2b0e141058321e69b91aaf3f3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f73a5e2b0e141058321e69b91aaf3f3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44709 } } }
I20260619 16:48:16.161044 21293 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8f73a5e2b0e141058321e69b91aaf3f3 [sys.catalog]: This master's current role is: LEADER
I20260619 16:48:16.161482 21292 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8f73a5e2b0e141058321e69b91aaf3f3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8f73a5e2b0e141058321e69b91aaf3f3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f73a5e2b0e141058321e69b91aaf3f3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44709 } } }
I20260619 16:48:16.161773 21292 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8f73a5e2b0e141058321e69b91aaf3f3 [sys.catalog]: This master's current role is: LEADER
I20260619 16:48:16.167126 21298 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:48:16.172189 21298 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:48:16.181144 21298 catalog_manager.cc:1347] Generated new cluster ID: c0b32ae9549e4dfa9fa80a12f569465c
I20260619 16:48:16.181263 21298 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:48:16.219864 21298 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 16:48:16.221207 21298 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:48:16.231611 21298 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8f73a5e2b0e141058321e69b91aaf3f3: Generated new TSK 0
I20260619 16:48:16.232431 21298 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 16:48:16.238655 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu pid 21214
W20260619 16:48:16.240088 21276 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:53805
W20260619 16:48:16.240751 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu pid 21214: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 16:48:16.240896 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu with pid 21214
2026-06-19T16:48:16Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1806 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781887385903558-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 16:48:16 dist-test-slave-wvz7 krb5kdc[21312](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 16:48:16 dist-test-slave-wvz7 krb5kdc[21312](info): set up 2 sockets
Jun 19 16:48:16 dist-test-slave-wvz7 krb5kdc[21312](info): commencing operation
krb5kdc: starting...
W20260619 16:48:17.254057 388 mini_kdc.cc:121] Time spent starting KDC: real 0.981s user 0.000s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781887385903558-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781887385903558-388-0/krb5kdc/test-user.keytab.
Jun 19 16:48:17 dist-test-slave-wvz7 krb5kdc[21312](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887697, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T16:48:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T16:48:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781887385903558-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781887385903558-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 16:48:17.494426 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu
/tmp/dist-test-taskEs2EH8/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781887385903558-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781887385903558-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781887385903558-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781887385903558-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40199
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781887385903558-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35791
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40199
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781887385903558-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781887385903558-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781887385903558-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:48:17.806078 21325 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:48:17.806344 21325 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:48:17.806439 21325 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:48:17.813323 21325 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:48:17.813426 21325 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:48:17.813478 21325 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:48:17.813539 21325 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:48:17.813580 21325 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:48:17.823089 21325 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35791
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781887385903558-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781887385903558-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40199
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781887385903558-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40199
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781887385903558-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781887385903558-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:48:17.824265 21325 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:48:17.825469 21325 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:48:17.836910 21331 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:48:17.836889 21333 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:48:17.836958 21330 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:48:17.837657 21325 server_base.cc:1034] running on GCE node
I20260619 16:48:17.838482 21325 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:48:17.839936 21325 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:48:17.841136 21325 hybrid_clock.cc:648] HybridClock initialized: now 1781887697841077 us; error 55 us; skew 500 ppm
I20260619 16:48:17.841467 21325 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:48:17 dist-test-slave-wvz7 krb5kdc[21312](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887697, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:48:17.844436 21325 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 16:48:17.845816 21325 webserver.cc:460] Webserver started at http://127.0.0.1:39827/ using document root <none> and password file <none>
I20260619 16:48:17.846449 21325 fs_manager.cc:359] Metadata directory not provided
I20260619 16:48:17.846546 21325 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:48:17.846783 21325 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:48:17.849534 21325 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781887385903558-388-0/minicluster-data/master-0/data/instance:
uuid: "ec3b1af4efc84902ae99f84f9373cc18"
format_stamp: "Formatted at 2026-06-19 16:48:17 on dist-test-slave-wvz7"
server_key: "068259e43670cc86a586d01fb28bc0c4"
server_key_iv: "af3d880df961eae6aafeb1813ce67336"
server_key_version: "encryptionkey@0"
I20260619 16:48:17.850255 21325 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781887385903558-388-0/minicluster-data/master-0/wal/instance:
uuid: "ec3b1af4efc84902ae99f84f9373cc18"
format_stamp: "Formatted at 2026-06-19 16:48:17 on dist-test-slave-wvz7"
server_key: "068259e43670cc86a586d01fb28bc0c4"
server_key_iv: "af3d880df961eae6aafeb1813ce67336"
server_key_version: "encryptionkey@0"
I20260619 16:48:17.855116 21325 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260619 16:48:17.858394 21340 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:48:17.859670 21325 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 16:48:17.859817 21325 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781887385903558-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781887385903558-388-0/minicluster-data/master-0/wal
uuid: "ec3b1af4efc84902ae99f84f9373cc18"
format_stamp: "Formatted at 2026-06-19 16:48:17 on dist-test-slave-wvz7"
server_key: "068259e43670cc86a586d01fb28bc0c4"
server_key_iv: "af3d880df961eae6aafeb1813ce67336"
server_key_version: "encryptionkey@0"
I20260619 16:48:17.860015 21325 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781887385903558-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781887385903558-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781887385903558-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:48:17.865363 21325 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:48:17.869189 21325 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:48:17.869463 21325 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:48:17.888017 21325 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40199
I20260619 16:48:17.888059 21391 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40199 every 8 connection(s)
I20260619 16:48:17.889863 21325 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781887385903558-388-0/minicluster-data/master-0/data/info.pb
I20260619 16:48:17.893522 21392 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:48:17.896005 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu as pid 21325
I20260619 16:48:17.896147 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781887385903558-388-0/minicluster-data/master-0/wal/instance
I20260619 16:48:17.896548 388 external_mini_cluster.cc:1428] Setting key 2ca873ce1c5ae6ac8facfa3598a1eaee
I20260619 16:48:17.905067 21395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:48:17.898250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43611 (local address 127.0.0.1:40199)
0619 16:48:17.899006 (+ 756us) server_negotiation.cc:206] Beginning negotiation
0619 16:48:17.899021 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 16:48:17.899069 (+ 48us) server_negotiation.cc:407] Connection header received
0619 16:48:17.901272 (+ 2203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:48:17.901300 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:48:17.901777 (+ 477us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:48:17.902214 (+ 437us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:48:17.903186 (+ 972us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:48:17.903195 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:48:17.903211 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:48:17.903228 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:48:17.903297 (+ 69us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:48:17.903328 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:48:17.903546 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0619 16:48:17.903673 (+ 127us) server_negotiation.cc:299] Negotiation successful
0619 16:48:17.903874 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":516,"thread_start_us":295,"threads_started":1}
I20260619 16:48:17.906579 21392 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ec3b1af4efc84902ae99f84f9373cc18: Bootstrap starting.
I20260619 16:48:17.908839 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 16:48:17.909685 21392 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ec3b1af4efc84902ae99f84f9373cc18: Neither blocks nor log segments found. Creating new log.
I20260619 16:48:17.910699 21392 log.cc:824] T 00000000000000000000000000000000 P ec3b1af4efc84902ae99f84f9373cc18: Log is configured to *not* fsync() on all Append() calls
I20260619 16:48:17.913705 21392 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ec3b1af4efc84902ae99f84f9373cc18: No bootstrap required, opened a new log
I20260619 16:48:17.915158 21395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:48:17.912916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36973 (local address 127.0.0.1:40199)
0619 16:48:17.913071 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0619 16:48:17.913077 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:48:17.913435 (+ 358us) server_negotiation.cc:407] Connection header received
0619 16:48:17.913583 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:48:17.913587 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:48:17.913632 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:48:17.913711 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:48:17.914573 (+ 862us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:48:17.914578 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:48:17.914581 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:48:17.914595 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:48:17.914631 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:48:17.914645 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:48:17.914742 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0619 16:48:17.914848 (+ 106us) server_negotiation.cc:299] Negotiation successful
0619 16:48:17.914905 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260619 16:48:17.921073 21392 raft_consensus.cc:348] T 00000000000000000000000000000000 P ec3b1af4efc84902ae99f84f9373cc18 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec3b1af4efc84902ae99f84f9373cc18" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40199 } }
I20260619 16:48:17.921506 21392 raft_consensus.cc:374] T 00000000000000000000000000000000 P ec3b1af4efc84902ae99f84f9373cc18 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:48:17.921623 21392 raft_consensus.cc:729] T 00000000000000000000000000000000 P ec3b1af4efc84902ae99f84f9373cc18 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec3b1af4efc84902ae99f84f9373cc18, State: Initialized, Role: FOLLOWER
I20260619 16:48:17.922207 21392 consensus_queue.cc:260] T 00000000000000000000000000000000 P ec3b1af4efc84902ae99f84f9373cc18 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec3b1af4efc84902ae99f84f9373cc18" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40199 } }
I20260619 16:48:17.922413 21392 raft_consensus.cc:388] T 00000000000000000000000000000000 P ec3b1af4efc84902ae99f84f9373cc18 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:48:17.922547 21392 raft_consensus.cc:482] T 00000000000000000000000000000000 P ec3b1af4efc84902ae99f84f9373cc18 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:48:17.922690 21392 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ec3b1af4efc84902ae99f84f9373cc18 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:48:17.924772 21392 raft_consensus.cc:504] T 00000000000000000000000000000000 P ec3b1af4efc84902ae99f84f9373cc18 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec3b1af4efc84902ae99f84f9373cc18" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40199 } }
I20260619 16:48:17.925240 21392 leader_election.cc:302] T 00000000000000000000000000000000 P ec3b1af4efc84902ae99f84f9373cc18 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ec3b1af4efc84902ae99f84f9373cc18; no voters:
I20260619 16:48:17.925794 21392 leader_election.cc:288] T 00000000000000000000000000000000 P ec3b1af4efc84902ae99f84f9373cc18 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 16:48:17.925925 21402 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ec3b1af4efc84902ae99f84f9373cc18 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:48:17.926523 21402 raft_consensus.cc:686] T 00000000000000000000000000000000 P ec3b1af4efc84902ae99f84f9373cc18 [term 1 LEADER]: Becoming Leader. State: Replica: ec3b1af4efc84902ae99f84f9373cc18, State: Running, Role: LEADER
I20260619 16:48:17.927102 21402 consensus_queue.cc:237] T 00000000000000000000000000000000 P ec3b1af4efc84902ae99f84f9373cc18 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec3b1af4efc84902ae99f84f9373cc18" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40199 } }
I20260619 16:48:17.927738 21392 sys_catalog.cc:564] T 00000000000000000000000000000000 P ec3b1af4efc84902ae99f84f9373cc18 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:48:17.929720 21403 sys_catalog.cc:455] T 00000000000000000000000000000000 P ec3b1af4efc84902ae99f84f9373cc18 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ec3b1af4efc84902ae99f84f9373cc18" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec3b1af4efc84902ae99f84f9373cc18" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40199 } } }
I20260619 16:48:17.929885 21404 sys_catalog.cc:455] T 00000000000000000000000000000000 P ec3b1af4efc84902ae99f84f9373cc18 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ec3b1af4efc84902ae99f84f9373cc18. Latest consensus state: current_term: 1 leader_uuid: "ec3b1af4efc84902ae99f84f9373cc18" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec3b1af4efc84902ae99f84f9373cc18" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40199 } } }
I20260619 16:48:17.930148 21404 sys_catalog.cc:458] T 00000000000000000000000000000000 P ec3b1af4efc84902ae99f84f9373cc18 [sys.catalog]: This master's current role is: LEADER
I20260619 16:48:17.930148 21403 sys_catalog.cc:458] T 00000000000000000000000000000000 P ec3b1af4efc84902ae99f84f9373cc18 [sys.catalog]: This master's current role is: LEADER
I20260619 16:48:17.933580 21411 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:48:17.938359 21411 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:48:17.947613 21411 catalog_manager.cc:1347] Generated new cluster ID: 409a0a7cbf0540709b73c2f70ee8d23f
I20260619 16:48:17.947734 21411 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:48:17.962349 21411 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 16:48:17.962805 21411 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:48:17.972229 21411 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ec3b1af4efc84902ae99f84f9373cc18: Generated new TSK 0
I20260619 16:48:17.973074 21411 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 16:48:17.987556 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu pid 21325
W20260619 16:48:17.988700 21387 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:43611
W20260619 16:48:17.989410 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu pid 21325: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 16:48:17.989560 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu with pid 21325
2026-06-19T16:48:18Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1750 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781887385903558-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 16:48:18 dist-test-slave-wvz7 krb5kdc[21423](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 16:48:18 dist-test-slave-wvz7 krb5kdc[21423](info): set up 2 sockets
Jun 19 16:48:18 dist-test-slave-wvz7 krb5kdc[21423](info): commencing operation
krb5kdc: starting...
W20260619 16:48:19.001959 388 mini_kdc.cc:121] Time spent starting KDC: real 0.978s user 0.000s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781887385903558-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781887385903558-388-0/krb5kdc/test-user.keytab.
Jun 19 16:48:19 dist-test-slave-wvz7 krb5kdc[21423](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887699, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T16:48:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T16:48:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781887385903558-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781887385903558-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 16:48:19.240885 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu
/tmp/dist-test-taskEs2EH8/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781887385903558-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781887385903558-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781887385903558-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781887385903558-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38547
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781887385903558-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45319
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38547
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781887385903558-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781887385903558-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781887385903558-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:48:19.565711 21436 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:48:19.566044 21436 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:48:19.566144 21436 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:48:19.573312 21436 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:48:19.573426 21436 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:48:19.573479 21436 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:48:19.573537 21436 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:48:19.573580 21436 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:48:19.583526 21436 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45319
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781887385903558-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781887385903558-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38547
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781887385903558-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38547
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781887385903558-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781887385903558-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:48:19.584785 21436 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:48:19.586069 21436 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:48:19.597450 21444 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:48:19.597518 21442 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:48:19.597486 21441 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:48:19.598378 21436 server_base.cc:1034] running on GCE node
I20260619 16:48:19.599189 21436 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:48:19.600701 21436 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:48:19.601902 21436 hybrid_clock.cc:648] HybridClock initialized: now 1781887699601857 us; error 42 us; skew 500 ppm
I20260619 16:48:19.602247 21436 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:48:19 dist-test-slave-wvz7 krb5kdc[21423](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887699, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:48:19.605211 21436 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 16:48:19.606768 21436 webserver.cc:460] Webserver started at http://127.0.0.1:42041/ using document root <none> and password file <none>
I20260619 16:48:19.607445 21436 fs_manager.cc:359] Metadata directory not provided
I20260619 16:48:19.607568 21436 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:48:19.607856 21436 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:48:19.610803 21436 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781887385903558-388-0/minicluster-data/master-0/data/instance:
uuid: "02185dcf900b4a5ca050c477e8e1b522"
format_stamp: "Formatted at 2026-06-19 16:48:19 on dist-test-slave-wvz7"
server_key: "5dab80140c282f1353f156a2a9d00e20"
server_key_iv: "02b03510182ec36d659fba35be16d171"
server_key_version: "encryptionkey@0"
I20260619 16:48:19.611533 21436 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781887385903558-388-0/minicluster-data/master-0/wal/instance:
uuid: "02185dcf900b4a5ca050c477e8e1b522"
format_stamp: "Formatted at 2026-06-19 16:48:19 on dist-test-slave-wvz7"
server_key: "5dab80140c282f1353f156a2a9d00e20"
server_key_iv: "02b03510182ec36d659fba35be16d171"
server_key_version: "encryptionkey@0"
I20260619 16:48:19.616706 21436 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260619 16:48:19.620071 21451 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:48:19.621409 21436 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260619 16:48:19.621577 21436 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781887385903558-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781887385903558-388-0/minicluster-data/master-0/wal
uuid: "02185dcf900b4a5ca050c477e8e1b522"
format_stamp: "Formatted at 2026-06-19 16:48:19 on dist-test-slave-wvz7"
server_key: "5dab80140c282f1353f156a2a9d00e20"
server_key_iv: "02b03510182ec36d659fba35be16d171"
server_key_version: "encryptionkey@0"
I20260619 16:48:19.621778 21436 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781887385903558-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781887385903558-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781887385903558-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:48:19.638226 21436 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:48:19.642262 21436 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:48:19.642539 21436 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:48:19.659209 21436 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38547
I20260619 16:48:19.659242 21502 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38547 every 8 connection(s)
I20260619 16:48:19.660544 21436 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781887385903558-388-0/minicluster-data/master-0/data/info.pb
I20260619 16:48:19.664175 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu as pid 21436
I20260619 16:48:19.664362 21503 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:48:19.664376 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781887385903558-388-0/minicluster-data/master-0/wal/instance
I20260619 16:48:19.664891 388 external_mini_cluster.cc:1428] Setting key 7781aa3e2602053979db7c8883fa240a
I20260619 16:48:19.678227 21503 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 02185dcf900b4a5ca050c477e8e1b522: Bootstrap starting.
Jun 19 16:48:19 dist-test-slave-wvz7 krb5kdc[21423](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887699, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260619 16:48:19.681200 21503 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 02185dcf900b4a5ca050c477e8e1b522: Neither blocks nor log segments found. Creating new log.
I20260619 16:48:19.682263 21503 log.cc:824] T 00000000000000000000000000000000 P 02185dcf900b4a5ca050c477e8e1b522: Log is configured to *not* fsync() on all Append() calls
I20260619 16:48:19.684785 21503 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 02185dcf900b4a5ca050c477e8e1b522: No bootstrap required, opened a new log
I20260619 16:48:19.686851 21506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:48:19.666640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.135:33621 (local address 127.0.0.1:38547)
0619 16:48:19.667406 (+ 766us) server_negotiation.cc:206] Beginning negotiation
0619 16:48:19.667418 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 16:48:19.667455 (+ 37us) server_negotiation.cc:407] Connection header received
0619 16:48:19.669035 (+ 1580us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:48:19.669056 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:48:19.669474 (+ 418us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:48:19.669832 (+ 358us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:48:19.671523 (+ 1691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:48:19.672808 (+ 1285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:48:19.674524 (+ 1716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:48:19.675237 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:48:19.675383 (+ 146us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:48:19.679852 (+ 4469us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:48:19.679861 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:48:19.679875 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:48:19.679919 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:48:19.682837 (+ 2918us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:48:19.683585 (+ 748us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:48:19.683591 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:48:19.683597 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:48:19.683755 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:48:19.684193 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:48:19.684197 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:48:19.684199 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:48:19.684688 (+ 489us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:48:19.684983 (+ 295us) server_negotiation.cc:1035] Waiting for connection context
0619 16:48:19.685493 (+ 510us) server_negotiation.cc:299] Negotiation successful
0619 16:48:19.685865 (+ 372us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":562,"thread_start_us":339,"threads_started":1}
I20260619 16:48:19.690980 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 16:48:19.693030 21503 raft_consensus.cc:348] T 00000000000000000000000000000000 P 02185dcf900b4a5ca050c477e8e1b522 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02185dcf900b4a5ca050c477e8e1b522" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38547 } }
I20260619 16:48:19.693444 21503 raft_consensus.cc:374] T 00000000000000000000000000000000 P 02185dcf900b4a5ca050c477e8e1b522 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:48:19.693553 21503 raft_consensus.cc:729] T 00000000000000000000000000000000 P 02185dcf900b4a5ca050c477e8e1b522 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02185dcf900b4a5ca050c477e8e1b522, State: Initialized, Role: FOLLOWER
I20260619 16:48:19.694253 21503 consensus_queue.cc:260] T 00000000000000000000000000000000 P 02185dcf900b4a5ca050c477e8e1b522 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02185dcf900b4a5ca050c477e8e1b522" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38547 } }
I20260619 16:48:19.694479 21503 raft_consensus.cc:388] T 00000000000000000000000000000000 P 02185dcf900b4a5ca050c477e8e1b522 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:48:19.694628 21503 raft_consensus.cc:482] T 00000000000000000000000000000000 P 02185dcf900b4a5ca050c477e8e1b522 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:48:19.694787 21503 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 02185dcf900b4a5ca050c477e8e1b522 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:48:19.697363 21503 raft_consensus.cc:504] T 00000000000000000000000000000000 P 02185dcf900b4a5ca050c477e8e1b522 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02185dcf900b4a5ca050c477e8e1b522" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38547 } }
I20260619 16:48:19.698015 21503 leader_election.cc:302] T 00000000000000000000000000000000 P 02185dcf900b4a5ca050c477e8e1b522 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 02185dcf900b4a5ca050c477e8e1b522; no voters:
I20260619 16:48:19.698637 21503 leader_election.cc:288] T 00000000000000000000000000000000 P 02185dcf900b4a5ca050c477e8e1b522 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 16:48:19.698740 21513 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 02185dcf900b4a5ca050c477e8e1b522 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:48:19.700591 21503 sys_catalog.cc:564] T 00000000000000000000000000000000 P 02185dcf900b4a5ca050c477e8e1b522 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:48:19.700555 21513 raft_consensus.cc:686] T 00000000000000000000000000000000 P 02185dcf900b4a5ca050c477e8e1b522 [term 1 LEADER]: Becoming Leader. State: Replica: 02185dcf900b4a5ca050c477e8e1b522, State: Running, Role: LEADER
I20260619 16:48:19.701153 21513 consensus_queue.cc:237] T 00000000000000000000000000000000 P 02185dcf900b4a5ca050c477e8e1b522 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02185dcf900b4a5ca050c477e8e1b522" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38547 } }
I20260619 16:48:19.703848 21515 sys_catalog.cc:455] T 00000000000000000000000000000000 P 02185dcf900b4a5ca050c477e8e1b522 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 02185dcf900b4a5ca050c477e8e1b522. Latest consensus state: current_term: 1 leader_uuid: "02185dcf900b4a5ca050c477e8e1b522" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02185dcf900b4a5ca050c477e8e1b522" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38547 } } }
I20260619 16:48:19.704231 21515 sys_catalog.cc:458] T 00000000000000000000000000000000 P 02185dcf900b4a5ca050c477e8e1b522 [sys.catalog]: This master's current role is: LEADER
I20260619 16:48:19.704468 21514 sys_catalog.cc:455] T 00000000000000000000000000000000 P 02185dcf900b4a5ca050c477e8e1b522 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "02185dcf900b4a5ca050c477e8e1b522" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02185dcf900b4a5ca050c477e8e1b522" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38547 } } }
I20260619 16:48:19.704748 21514 sys_catalog.cc:458] T 00000000000000000000000000000000 P 02185dcf900b4a5ca050c477e8e1b522 [sys.catalog]: This master's current role is: LEADER
I20260619 16:48:19.705641 21520 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:48:19.710692 21520 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:48:19.712963 21506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:48:19.695102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.135:48749 (local address 127.0.0.1:38547)
0619 16:48:19.695246 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0619 16:48:19.695250 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:48:19.695711 (+ 461us) server_negotiation.cc:407] Connection header received
0619 16:48:19.695836 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:48:19.695841 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:48:19.695932 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:48:19.696035 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:48:19.697369 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:48:19.698494 (+ 1125us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:48:19.700336 (+ 1842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:48:19.701027 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:48:19.701160 (+ 133us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:48:19.704200 (+ 3040us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:48:19.704208 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:48:19.704213 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:48:19.704258 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:48:19.708383 (+ 4125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:48:19.709491 (+ 1108us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:48:19.709496 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:48:19.709500 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:48:19.709655 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:48:19.710245 (+ 590us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:48:19.710250 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:48:19.710254 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:48:19.710778 (+ 524us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:48:19.712179 (+ 1401us) server_negotiation.cc:1035] Waiting for connection context
0619 16:48:19.712370 (+ 191us) server_negotiation.cc:299] Negotiation successful
0619 16:48:19.712570 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260619 16:48:19.725518 21520 catalog_manager.cc:1347] Generated new cluster ID: 4ce851e7ccb74588984321f7992c9e4f
I20260619 16:48:19.725654 21520 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:48:19.748787 21520 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 16:48:19.750150 21520 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:48:19.756592 21520 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 02185dcf900b4a5ca050c477e8e1b522: Generated new TSK 0
I20260619 16:48:19.757417 21520 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 16:48:19.769173 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu pid 21436
I20260619 16:48:19.770483 21498 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:48:19.861445 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu with pid 21436
W20260619 16:48:19.875080 21421 connection.cc:535] client connection to 127.0.0.1:38547 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:38547): Connection reset by peer (error 104)
2026-06-19T16:48:19Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1871 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781887385903558-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 16:48:19 dist-test-slave-wvz7 krb5kdc[21535](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 16:48:19 dist-test-slave-wvz7 krb5kdc[21535](info): set up 2 sockets
Jun 19 16:48:19 dist-test-slave-wvz7 krb5kdc[21535](info): commencing operation
krb5kdc: starting...
W20260619 16:48:20.877889 388 mini_kdc.cc:121] Time spent starting KDC: real 0.981s user 0.002s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781887385903558-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781887385903558-388-0/krb5kdc/test-user.keytab.
Jun 19 16:48:21 dist-test-slave-wvz7 krb5kdc[21535](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887701, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T16:48:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T16:48:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781887385903558-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781887385903558-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 16:48:21.124421 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu
/tmp/dist-test-taskEs2EH8/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781887385903558-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781887385903558-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781887385903558-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781887385903558-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45127
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781887385903558-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34767
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45127
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781887385903558-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781887385903558-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781887385903558-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:48:21.444589 21548 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:48:21.444914 21548 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:48:21.445053 21548 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:48:21.452575 21548 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:48:21.452754 21548 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:48:21.452844 21548 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:48:21.452932 21548 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:48:21.452999 21548 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:48:21.463218 21548 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34767
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781887385903558-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781887385903558-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45127
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781887385903558-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45127
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781887385903558-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781887385903558-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:48:21.464589 21548 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:48:21.466032 21548 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:48:21.478171 21554 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:48:21.478277 21553 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:48:21.478228 21556 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:48:21.478880 21548 server_base.cc:1034] running on GCE node
I20260619 16:48:21.479689 21548 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:48:21.481159 21548 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:48:21.482378 21548 hybrid_clock.cc:648] HybridClock initialized: now 1781887701482318 us; error 56 us; skew 500 ppm
I20260619 16:48:21.482748 21548 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:48:21 dist-test-slave-wvz7 krb5kdc[21535](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887701, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:48:21.491665 21548 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 16:48:21.493109 21548 webserver.cc:460] Webserver started at http://127.0.0.1:43041/ using document root <none> and password file <none>
I20260619 16:48:21.493793 21548 fs_manager.cc:359] Metadata directory not provided
I20260619 16:48:21.493912 21548 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:48:21.494182 21548 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:48:21.497128 21548 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781887385903558-388-0/minicluster-data/master-0/data/instance:
uuid: "9e090141a2b944b395bcf287270480b0"
format_stamp: "Formatted at 2026-06-19 16:48:21 on dist-test-slave-wvz7"
server_key: "79113e2d4c4dfd36acde7dfa8d38d1e1"
server_key_iv: "a637628b4bd937feda349ac61c1bdd83"
server_key_version: "encryptionkey@0"
I20260619 16:48:21.497929 21548 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781887385903558-388-0/minicluster-data/master-0/wal/instance:
uuid: "9e090141a2b944b395bcf287270480b0"
format_stamp: "Formatted at 2026-06-19 16:48:21 on dist-test-slave-wvz7"
server_key: "79113e2d4c4dfd36acde7dfa8d38d1e1"
server_key_iv: "a637628b4bd937feda349ac61c1bdd83"
server_key_version: "encryptionkey@0"
I20260619 16:48:21.502969 21548 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260619 16:48:21.506573 21563 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:48:21.507882 21548 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 16:48:21.508019 21548 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781887385903558-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781887385903558-388-0/minicluster-data/master-0/wal
uuid: "9e090141a2b944b395bcf287270480b0"
format_stamp: "Formatted at 2026-06-19 16:48:21 on dist-test-slave-wvz7"
server_key: "79113e2d4c4dfd36acde7dfa8d38d1e1"
server_key_iv: "a637628b4bd937feda349ac61c1bdd83"
server_key_version: "encryptionkey@0"
I20260619 16:48:21.508172 21548 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781887385903558-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781887385903558-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781887385903558-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:48:21.524178 21548 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:48:21.528389 21548 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:48:21.528622 21548 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:48:21.546142 21548 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45127
I20260619 16:48:21.546181 21614 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45127 every 8 connection(s)
I20260619 16:48:21.547492 21548 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781887385903558-388-0/minicluster-data/master-0/data/info.pb
I20260619 16:48:21.551234 21615 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:48:21.557281 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu as pid 21548
I20260619 16:48:21.557461 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781887385903558-388-0/minicluster-data/master-0/wal/instance
I20260619 16:48:21.557940 388 external_mini_cluster.cc:1428] Setting key 533b14076667d71c86f457d0a712fbcb
I20260619 16:48:21.561866 21615 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9e090141a2b944b395bcf287270480b0: Bootstrap starting.
I20260619 16:48:21.565176 21615 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9e090141a2b944b395bcf287270480b0: Neither blocks nor log segments found. Creating new log.
I20260619 16:48:21.566432 21615 log.cc:824] T 00000000000000000000000000000000 P 9e090141a2b944b395bcf287270480b0: Log is configured to *not* fsync() on all Append() calls
I20260619 16:48:21.569655 21615 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9e090141a2b944b395bcf287270480b0: No bootstrap required, opened a new log
I20260619 16:48:21.571661 21618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:48:21.559819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.135:36203 (local address 127.0.0.1:45127)
0619 16:48:21.560547 (+ 728us) server_negotiation.cc:206] Beginning negotiation
0619 16:48:21.560560 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 16:48:21.560607 (+ 47us) server_negotiation.cc:407] Connection header received
0619 16:48:21.562377 (+ 1770us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:48:21.562407 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:48:21.562827 (+ 420us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:48:21.563147 (+ 320us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:48:21.563785 (+ 638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:48:21.565048 (+ 1263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:48:21.566893 (+ 1845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:48:21.567956 (+ 1063us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:48:21.568084 (+ 128us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:48:21.569329 (+ 1245us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:48:21.569336 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:48:21.569351 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:48:21.569946 (+ 595us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:48:21.570035 (+ 89us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:48:21.570060 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:48:21.570180 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0619 16:48:21.570508 (+ 328us) server_negotiation.cc:299] Negotiation successful
0619 16:48:21.570697 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":512,"thread_start_us":310,"threads_started":1}
I20260619 16:48:21.575358 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 16:48:21.577325 21615 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9e090141a2b944b395bcf287270480b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e090141a2b944b395bcf287270480b0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45127 } }
I20260619 16:48:21.577703 21615 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9e090141a2b944b395bcf287270480b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:48:21.577831 21615 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9e090141a2b944b395bcf287270480b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e090141a2b944b395bcf287270480b0, State: Initialized, Role: FOLLOWER
I20260619 16:48:21.578598 21615 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9e090141a2b944b395bcf287270480b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e090141a2b944b395bcf287270480b0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45127 } }
I20260619 16:48:21.578821 21615 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9e090141a2b944b395bcf287270480b0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:48:21.578946 21615 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9e090141a2b944b395bcf287270480b0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:48:21.579093 21615 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9e090141a2b944b395bcf287270480b0 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:48:21.581724 21615 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9e090141a2b944b395bcf287270480b0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e090141a2b944b395bcf287270480b0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45127 } }
I20260619 16:48:21.582253 21615 leader_election.cc:302] T 00000000000000000000000000000000 P 9e090141a2b944b395bcf287270480b0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9e090141a2b944b395bcf287270480b0; no voters:
I20260619 16:48:21.582806 21615 leader_election.cc:288] T 00000000000000000000000000000000 P 9e090141a2b944b395bcf287270480b0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 16:48:21.582950 21625 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9e090141a2b944b395bcf287270480b0 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:48:21.584244 21625 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9e090141a2b944b395bcf287270480b0 [term 1 LEADER]: Becoming Leader. State: Replica: 9e090141a2b944b395bcf287270480b0, State: Running, Role: LEADER
I20260619 16:48:21.584846 21615 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9e090141a2b944b395bcf287270480b0 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:48:21.584851 21625 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9e090141a2b944b395bcf287270480b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e090141a2b944b395bcf287270480b0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45127 } }
I20260619 16:48:21.587245 21618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:48:21.579992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.135:37137 (local address 127.0.0.1:45127)
0619 16:48:21.580154 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0619 16:48:21.580158 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:48:21.580617 (+ 459us) server_negotiation.cc:407] Connection header received
0619 16:48:21.580794 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:48:21.580798 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:48:21.580859 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:48:21.580942 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:48:21.581437 (+ 495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:48:21.582246 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:48:21.584027 (+ 1781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:48:21.584754 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:48:21.584921 (+ 167us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:48:21.586176 (+ 1255us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:48:21.586182 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:48:21.586186 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:48:21.586207 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:48:21.586291 (+ 84us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:48:21.586315 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:48:21.586432 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0619 16:48:21.586857 (+ 425us) server_negotiation.cc:299] Negotiation successful
0619 16:48:21.586914 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260619 16:48:21.587596 21627 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9e090141a2b944b395bcf287270480b0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9e090141a2b944b395bcf287270480b0. Latest consensus state: current_term: 1 leader_uuid: "9e090141a2b944b395bcf287270480b0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e090141a2b944b395bcf287270480b0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45127 } } }
I20260619 16:48:21.587972 21627 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9e090141a2b944b395bcf287270480b0 [sys.catalog]: This master's current role is: LEADER
I20260619 16:48:21.590417 21626 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9e090141a2b944b395bcf287270480b0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9e090141a2b944b395bcf287270480b0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e090141a2b944b395bcf287270480b0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45127 } } }
I20260619 16:48:21.590729 21626 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9e090141a2b944b395bcf287270480b0 [sys.catalog]: This master's current role is: LEADER
I20260619 16:48:21.594300 21631 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:48:21.599404 21631 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:48:21.607780 21631 catalog_manager.cc:1347] Generated new cluster ID: 237c775cf6f7455e8faa11d97b88f534
I20260619 16:48:21.607901 21631 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:48:21.632521 21631 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 16:48:21.633801 21631 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:48:21.643218 21631 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9e090141a2b944b395bcf287270480b0: Generated new TSK 0
I20260619 16:48:21.644042 21631 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 16:48:21.655483 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu pid 21548
W20260619 16:48:21.656563 21610 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.135:36203
W20260619 16:48:21.657323 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu pid 21548: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 16:48:21.657472 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu with pid 21548
W20260619 16:48:21.671164 21533 connection.cc:535] client connection to 127.0.0.1:45127 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:45127): Connection reset by peer (error 104)
2026-06-19T16:48:21Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1794 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781887385903558-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 16:48:21 dist-test-slave-wvz7 krb5kdc[21646](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 16:48:21 dist-test-slave-wvz7 krb5kdc[21646](info): set up 2 sockets
Jun 19 16:48:21 dist-test-slave-wvz7 krb5kdc[21646](info): commencing operation
krb5kdc: starting...
W20260619 16:48:22.669898 388 mini_kdc.cc:121] Time spent starting KDC: real 0.980s user 0.003s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781887385903558-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781887385903558-388-0/krb5kdc/test-user.keytab.
Jun 19 16:48:22 dist-test-slave-wvz7 krb5kdc[21646](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887702, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T16:48:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T16:48:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781887385903558-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781887385903558-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 16:48:22.906384 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu
/tmp/dist-test-taskEs2EH8/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781887385903558-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781887385903558-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781887385903558-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781887385903558-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39671
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781887385903558-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35807
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39671
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781887385903558-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781887385903558-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781887385903558-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:48:23.218253 21659 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:48:23.218523 21659 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:48:23.218619 21659 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:48:23.225589 21659 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:48:23.225704 21659 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:48:23.225787 21659 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:48:23.225857 21659 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:48:23.225899 21659 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:48:23.235493 21659 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35807
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781887385903558-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781887385903558-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39671
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781887385903558-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39671
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781887385903558-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781887385903558-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:48:23.236691 21659 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:48:23.237998 21659 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:48:23.249212 21665 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:48:23.249289 21664 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:48:23.249235 21667 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:48:23.249698 21659 server_base.cc:1034] running on GCE node
I20260619 16:48:23.250566 21659 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:48:23.251991 21659 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:48:23.253216 21659 hybrid_clock.cc:648] HybridClock initialized: now 1781887703253162 us; error 41 us; skew 500 ppm
I20260619 16:48:23.253556 21659 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:48:23 dist-test-slave-wvz7 krb5kdc[21646](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887703, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:48:23.256637 21659 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 16:48:23.258129 21659 webserver.cc:460] Webserver started at http://127.0.0.1:33853/ using document root <none> and password file <none>
I20260619 16:48:23.258776 21659 fs_manager.cc:359] Metadata directory not provided
I20260619 16:48:23.258883 21659 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:48:23.259158 21659 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:48:23.262073 21659 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781887385903558-388-0/minicluster-data/master-0/data/instance:
uuid: "591f245082eb4f06a8c72672b36d2fc9"
format_stamp: "Formatted at 2026-06-19 16:48:23 on dist-test-slave-wvz7"
server_key: "36dbe528ebcd4363754c95b693164d08"
server_key_iv: "dc12a37dfecb067b2b97c98d99610312"
server_key_version: "encryptionkey@0"
I20260619 16:48:23.262831 21659 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781887385903558-388-0/minicluster-data/master-0/wal/instance:
uuid: "591f245082eb4f06a8c72672b36d2fc9"
format_stamp: "Formatted at 2026-06-19 16:48:23 on dist-test-slave-wvz7"
server_key: "36dbe528ebcd4363754c95b693164d08"
server_key_iv: "dc12a37dfecb067b2b97c98d99610312"
server_key_version: "encryptionkey@0"
I20260619 16:48:23.268024 21659 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 16:48:23.271857 21674 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:48:23.273253 21659 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260619 16:48:23.273437 21659 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781887385903558-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781887385903558-388-0/minicluster-data/master-0/wal
uuid: "591f245082eb4f06a8c72672b36d2fc9"
format_stamp: "Formatted at 2026-06-19 16:48:23 on dist-test-slave-wvz7"
server_key: "36dbe528ebcd4363754c95b693164d08"
server_key_iv: "dc12a37dfecb067b2b97c98d99610312"
server_key_version: "encryptionkey@0"
I20260619 16:48:23.273636 21659 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781887385903558-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781887385903558-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781887385903558-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:48:23.279273 21659 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:48:23.283182 21659 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:48:23.283442 21659 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:48:23.300554 21659 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39671
I20260619 16:48:23.300587 21725 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39671 every 8 connection(s)
I20260619 16:48:23.301942 21659 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781887385903558-388-0/minicluster-data/master-0/data/info.pb
I20260619 16:48:23.305550 21726 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:48:23.308327 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu as pid 21659
I20260619 16:48:23.308504 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781887385903558-388-0/minicluster-data/master-0/wal/instance
I20260619 16:48:23.308982 388 external_mini_cluster.cc:1428] Setting key 1cf1cf02c1e769495f66bf9cb93c6722
I20260619 16:48:23.317027 21729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:48:23.310839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.135:57099 (local address 127.0.0.1:39671)
0619 16:48:23.311605 (+ 766us) server_negotiation.cc:206] Beginning negotiation
0619 16:48:23.311617 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 16:48:23.311654 (+ 37us) server_negotiation.cc:407] Connection header received
0619 16:48:23.313303 (+ 1649us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:48:23.313325 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:48:23.313667 (+ 342us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:48:23.314098 (+ 431us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:48:23.315090 (+ 992us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:48:23.315096 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:48:23.315108 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:48:23.315121 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:48:23.315180 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:48:23.315199 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:48:23.315409 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0619 16:48:23.315535 (+ 126us) server_negotiation.cc:299] Negotiation successful
0619 16:48:23.315744 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":573,"thread_start_us":341,"threads_started":1}
I20260619 16:48:23.317940 21726 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 591f245082eb4f06a8c72672b36d2fc9: Bootstrap starting.
I20260619 16:48:23.321419 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 16:48:23.321764 21726 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 591f245082eb4f06a8c72672b36d2fc9: Neither blocks nor log segments found. Creating new log.
I20260619 16:48:23.322976 21726 log.cc:824] T 00000000000000000000000000000000 P 591f245082eb4f06a8c72672b36d2fc9: Log is configured to *not* fsync() on all Append() calls
I20260619 16:48:23.326133 21726 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 591f245082eb4f06a8c72672b36d2fc9: No bootstrap required, opened a new log
I20260619 16:48:23.328940 21729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:48:23.326669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.135:38821 (local address 127.0.0.1:39671)
0619 16:48:23.326862 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0619 16:48:23.326867 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:48:23.327166 (+ 299us) server_negotiation.cc:407] Connection header received
0619 16:48:23.327341 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:48:23.327344 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:48:23.327388 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:48:23.327462 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:48:23.328335 (+ 873us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:48:23.328339 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:48:23.328342 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:48:23.328369 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:48:23.328405 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:48:23.328419 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:48:23.328510 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0619 16:48:23.328621 (+ 111us) server_negotiation.cc:299] Negotiation successful
0619 16:48:23.328667 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260619 16:48:23.334004 21726 raft_consensus.cc:348] T 00000000000000000000000000000000 P 591f245082eb4f06a8c72672b36d2fc9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "591f245082eb4f06a8c72672b36d2fc9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39671 } }
I20260619 16:48:23.334393 21726 raft_consensus.cc:374] T 00000000000000000000000000000000 P 591f245082eb4f06a8c72672b36d2fc9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:48:23.334481 21726 raft_consensus.cc:729] T 00000000000000000000000000000000 P 591f245082eb4f06a8c72672b36d2fc9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 591f245082eb4f06a8c72672b36d2fc9, State: Initialized, Role: FOLLOWER
I20260619 16:48:23.335090 21726 consensus_queue.cc:260] T 00000000000000000000000000000000 P 591f245082eb4f06a8c72672b36d2fc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "591f245082eb4f06a8c72672b36d2fc9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39671 } }
I20260619 16:48:23.335325 21726 raft_consensus.cc:388] T 00000000000000000000000000000000 P 591f245082eb4f06a8c72672b36d2fc9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:48:23.335449 21726 raft_consensus.cc:482] T 00000000000000000000000000000000 P 591f245082eb4f06a8c72672b36d2fc9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:48:23.335592 21726 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 591f245082eb4f06a8c72672b36d2fc9 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:48:23.337694 21726 raft_consensus.cc:504] T 00000000000000000000000000000000 P 591f245082eb4f06a8c72672b36d2fc9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "591f245082eb4f06a8c72672b36d2fc9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39671 } }
I20260619 16:48:23.338231 21726 leader_election.cc:302] T 00000000000000000000000000000000 P 591f245082eb4f06a8c72672b36d2fc9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 591f245082eb4f06a8c72672b36d2fc9; no voters:
I20260619 16:48:23.338781 21726 leader_election.cc:288] T 00000000000000000000000000000000 P 591f245082eb4f06a8c72672b36d2fc9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 16:48:23.338935 21736 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 591f245082eb4f06a8c72672b36d2fc9 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:48:23.339560 21736 raft_consensus.cc:686] T 00000000000000000000000000000000 P 591f245082eb4f06a8c72672b36d2fc9 [term 1 LEADER]: Becoming Leader. State: Replica: 591f245082eb4f06a8c72672b36d2fc9, State: Running, Role: LEADER
I20260619 16:48:23.340178 21736 consensus_queue.cc:237] T 00000000000000000000000000000000 P 591f245082eb4f06a8c72672b36d2fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "591f245082eb4f06a8c72672b36d2fc9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39671 } }
I20260619 16:48:23.341302 21726 sys_catalog.cc:564] T 00000000000000000000000000000000 P 591f245082eb4f06a8c72672b36d2fc9 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:48:23.342770 21738 sys_catalog.cc:455] T 00000000000000000000000000000000 P 591f245082eb4f06a8c72672b36d2fc9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "591f245082eb4f06a8c72672b36d2fc9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "591f245082eb4f06a8c72672b36d2fc9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39671 } } }
I20260619 16:48:23.343165 21738 sys_catalog.cc:458] T 00000000000000000000000000000000 P 591f245082eb4f06a8c72672b36d2fc9 [sys.catalog]: This master's current role is: LEADER
I20260619 16:48:23.343611 21737 sys_catalog.cc:455] T 00000000000000000000000000000000 P 591f245082eb4f06a8c72672b36d2fc9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 591f245082eb4f06a8c72672b36d2fc9. Latest consensus state: current_term: 1 leader_uuid: "591f245082eb4f06a8c72672b36d2fc9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "591f245082eb4f06a8c72672b36d2fc9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39671 } } }
I20260619 16:48:23.343962 21737 sys_catalog.cc:458] T 00000000000000000000000000000000 P 591f245082eb4f06a8c72672b36d2fc9 [sys.catalog]: This master's current role is: LEADER
I20260619 16:48:23.344610 21743 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:48:23.352663 21743 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:48:23.360930 21743 catalog_manager.cc:1347] Generated new cluster ID: 66ade320400b40f9846b1d2146f77a78
I20260619 16:48:23.361043 21743 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:48:23.375738 21743 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 16:48:23.376161 21743 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:48:23.386363 21743 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 591f245082eb4f06a8c72672b36d2fc9: Generated new TSK 0
I20260619 16:48:23.387241 21743 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 16:48:23.402719 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu pid 21659
W20260619 16:48:23.403960 21721 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.135:57099
W20260619 16:48:23.404609 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu pid 21659: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 16:48:23.404732 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu with pid 21659
2026-06-19T16:48:23Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1746 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781887385903558-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 16:48:23 dist-test-slave-wvz7 krb5kdc[21757](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 16:48:23 dist-test-slave-wvz7 krb5kdc[21757](info): set up 2 sockets
Jun 19 16:48:23 dist-test-slave-wvz7 krb5kdc[21757](info): commencing operation
krb5kdc: starting...
W20260619 16:48:24.421942 388 mini_kdc.cc:121] Time spent starting KDC: real 0.985s user 0.000s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781887385903558-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781887385903558-388-0/krb5kdc/test-user.keytab.
Jun 19 16:48:24 dist-test-slave-wvz7 krb5kdc[21757](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887704, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T16:48:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T16:48:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781887385903558-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781887385903558-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 16:48:24.652890 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu
/tmp/dist-test-taskEs2EH8/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781887385903558-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781887385903558-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781887385903558-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781887385903558-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38795
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781887385903558-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42301
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38795
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781887385903558-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781887385903558-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781887385903558-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:48:24.962008 21770 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:48:24.962355 21770 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:48:24.962512 21770 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:48:24.969805 21770 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:48:24.969910 21770 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:48:24.969960 21770 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:48:24.970018 21770 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:48:24.970059 21770 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:48:24.979713 21770 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42301
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781887385903558-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781887385903558-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38795
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781887385903558-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38795
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781887385903558-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781887385903558-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:48:24.980928 21770 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:48:24.982240 21770 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:48:24.993655 21775 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:48:24.993697 21776 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:48:24.993894 21778 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:48:24.994554 21770 server_base.cc:1034] running on GCE node
I20260619 16:48:24.995297 21770 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:48:24.996614 21770 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:48:24.997879 21770 hybrid_clock.cc:648] HybridClock initialized: now 1781887704997832 us; error 28 us; skew 500 ppm
I20260619 16:48:24.998227 21770 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:48:24 dist-test-slave-wvz7 krb5kdc[21757](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887704, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:48:25.001327 21770 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 16:48:25.002787 21770 webserver.cc:460] Webserver started at http://127.0.0.1:40073/ using document root <none> and password file <none>
I20260619 16:48:25.003491 21770 fs_manager.cc:359] Metadata directory not provided
I20260619 16:48:25.003603 21770 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:48:25.003881 21770 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:48:25.006831 21770 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781887385903558-388-0/minicluster-data/master-0/data/instance:
uuid: "04e20eba6005411cafece401b127948f"
format_stamp: "Formatted at 2026-06-19 16:48:24 on dist-test-slave-wvz7"
server_key: "23a511dfcfbccec29501f5aa2c502b09"
server_key_iv: "1ae2d84b54d2daf6f391ac86ef7189a2"
server_key_version: "encryptionkey@0"
I20260619 16:48:25.007553 21770 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781887385903558-388-0/minicluster-data/master-0/wal/instance:
uuid: "04e20eba6005411cafece401b127948f"
format_stamp: "Formatted at 2026-06-19 16:48:24 on dist-test-slave-wvz7"
server_key: "23a511dfcfbccec29501f5aa2c502b09"
server_key_iv: "1ae2d84b54d2daf6f391ac86ef7189a2"
server_key_version: "encryptionkey@0"
I20260619 16:48:25.012552 21770 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260619 16:48:25.016014 21785 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:48:25.017330 21770 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260619 16:48:25.017488 21770 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781887385903558-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781887385903558-388-0/minicluster-data/master-0/wal
uuid: "04e20eba6005411cafece401b127948f"
format_stamp: "Formatted at 2026-06-19 16:48:24 on dist-test-slave-wvz7"
server_key: "23a511dfcfbccec29501f5aa2c502b09"
server_key_iv: "1ae2d84b54d2daf6f391ac86ef7189a2"
server_key_version: "encryptionkey@0"
I20260619 16:48:25.017681 21770 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781887385903558-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781887385903558-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781887385903558-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:48:25.033084 21770 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:48:25.037318 21770 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:48:25.037570 21770 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:48:25.054728 21770 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38795
I20260619 16:48:25.054749 21836 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38795 every 8 connection(s)
I20260619 16:48:25.056054 21770 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781887385903558-388-0/minicluster-data/master-0/data/info.pb
I20260619 16:48:25.059713 21837 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:48:25.064484 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu as pid 21770
I20260619 16:48:25.064639 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781887385903558-388-0/minicluster-data/master-0/wal/instance
I20260619 16:48:25.065032 388 external_mini_cluster.cc:1428] Setting key 098f3bf5e596e4e8bf2bdf80067a0123
I20260619 16:48:25.071477 21837 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 04e20eba6005411cafece401b127948f: Bootstrap starting.
I20260619 16:48:25.075069 21837 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 04e20eba6005411cafece401b127948f: Neither blocks nor log segments found. Creating new log.
I20260619 16:48:25.076083 21837 log.cc:824] T 00000000000000000000000000000000 P 04e20eba6005411cafece401b127948f: Log is configured to *not* fsync() on all Append() calls
I20260619 16:48:25.079162 21837 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 04e20eba6005411cafece401b127948f: No bootstrap required, opened a new log
Jun 19 16:48:25 dist-test-slave-wvz7 krb5kdc[21757](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887704, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260619 16:48:25.088120 21837 raft_consensus.cc:348] T 00000000000000000000000000000000 P 04e20eba6005411cafece401b127948f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04e20eba6005411cafece401b127948f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38795 } }
I20260619 16:48:25.088475 21837 raft_consensus.cc:374] T 00000000000000000000000000000000 P 04e20eba6005411cafece401b127948f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:48:25.088587 21837 raft_consensus.cc:729] T 00000000000000000000000000000000 P 04e20eba6005411cafece401b127948f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 04e20eba6005411cafece401b127948f, State: Initialized, Role: FOLLOWER
I20260619 16:48:25.089126 21837 consensus_queue.cc:260] T 00000000000000000000000000000000 P 04e20eba6005411cafece401b127948f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04e20eba6005411cafece401b127948f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38795 } }
I20260619 16:48:25.089344 21840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:48:25.066918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.135:39923 (local address 127.0.0.1:38795)
0619 16:48:25.067678 (+ 760us) server_negotiation.cc:206] Beginning negotiation
0619 16:48:25.067693 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 16:48:25.067758 (+ 65us) server_negotiation.cc:407] Connection header received
0619 16:48:25.070174 (+ 2416us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:48:25.070203 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:48:25.070719 (+ 516us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:48:25.071177 (+ 458us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:48:25.072518 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:48:25.074518 (+ 2000us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:48:25.075867 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:48:25.076874 (+ 1007us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:48:25.077003 (+ 129us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:48:25.081696 (+ 4693us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:48:25.081704 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:48:25.081718 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:48:25.081801 (+ 83us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:48:25.084808 (+ 3007us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:48:25.085806 (+ 998us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:48:25.085813 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:48:25.085820 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:48:25.085995 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:48:25.086448 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:48:25.086451 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:48:25.086453 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:48:25.087028 (+ 575us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:48:25.087263 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0619 16:48:25.087789 (+ 526us) server_negotiation.cc:299] Negotiation successful
0619 16:48:25.088101 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":507,"thread_start_us":300,"threads_started":1}
I20260619 16:48:25.089349 21837 raft_consensus.cc:388] T 00000000000000000000000000000000 P 04e20eba6005411cafece401b127948f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:48:25.089521 21837 raft_consensus.cc:482] T 00000000000000000000000000000000 P 04e20eba6005411cafece401b127948f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:48:25.089694 21837 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 04e20eba6005411cafece401b127948f [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:48:25.092046 21837 raft_consensus.cc:504] T 00000000000000000000000000000000 P 04e20eba6005411cafece401b127948f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04e20eba6005411cafece401b127948f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38795 } }
I20260619 16:48:25.092561 21837 leader_election.cc:302] T 00000000000000000000000000000000 P 04e20eba6005411cafece401b127948f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 04e20eba6005411cafece401b127948f; no voters:
I20260619 16:48:25.093108 21837 leader_election.cc:288] T 00000000000000000000000000000000 P 04e20eba6005411cafece401b127948f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 16:48:25.093138 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 16:48:25.093235 21842 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 04e20eba6005411cafece401b127948f [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:48:25.094923 21837 sys_catalog.cc:564] T 00000000000000000000000000000000 P 04e20eba6005411cafece401b127948f [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:48:25.099514 21842 raft_consensus.cc:686] T 00000000000000000000000000000000 P 04e20eba6005411cafece401b127948f [term 1 LEADER]: Becoming Leader. State: Replica: 04e20eba6005411cafece401b127948f, State: Running, Role: LEADER
I20260619 16:48:25.100090 21842 consensus_queue.cc:237] T 00000000000000000000000000000000 P 04e20eba6005411cafece401b127948f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04e20eba6005411cafece401b127948f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38795 } }
I20260619 16:48:25.102907 21844 sys_catalog.cc:455] T 00000000000000000000000000000000 P 04e20eba6005411cafece401b127948f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 04e20eba6005411cafece401b127948f. Latest consensus state: current_term: 1 leader_uuid: "04e20eba6005411cafece401b127948f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04e20eba6005411cafece401b127948f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38795 } } }
I20260619 16:48:25.103281 21844 sys_catalog.cc:458] T 00000000000000000000000000000000 P 04e20eba6005411cafece401b127948f [sys.catalog]: This master's current role is: LEADER
I20260619 16:48:25.102974 21843 sys_catalog.cc:455] T 00000000000000000000000000000000 P 04e20eba6005411cafece401b127948f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "04e20eba6005411cafece401b127948f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04e20eba6005411cafece401b127948f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38795 } } }
I20260619 16:48:25.103677 21843 sys_catalog.cc:458] T 00000000000000000000000000000000 P 04e20eba6005411cafece401b127948f [sys.catalog]: This master's current role is: LEADER
W20260619 16:48:25.110085 21862 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 04e20eba6005411cafece401b127948f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 16:48:25.110203 21862 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 16:48:25.110340 21861 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:48:25.112202 21861 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:48:25.118350 21840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:48:25.101361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.135:41293 (local address 127.0.0.1:38795)
0619 16:48:25.101531 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0619 16:48:25.101538 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:48:25.102121 (+ 583us) server_negotiation.cc:407] Connection header received
0619 16:48:25.102364 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:48:25.102370 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:48:25.102454 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:48:25.102524 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:48:25.104539 (+ 2015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:48:25.105883 (+ 1344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:48:25.107518 (+ 1635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:48:25.108190 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:48:25.108352 (+ 162us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:48:25.111285 (+ 2933us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:48:25.111292 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:48:25.111297 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:48:25.111335 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:48:25.115075 (+ 3740us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:48:25.116037 (+ 962us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:48:25.116042 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:48:25.116045 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:48:25.116179 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:48:25.116711 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:48:25.116716 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:48:25.116719 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:48:25.117113 (+ 394us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:48:25.117355 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0619 16:48:25.117841 (+ 486us) server_negotiation.cc:299] Negotiation successful
0619 16:48:25.118016 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260619 16:48:25.121465 21861 catalog_manager.cc:1347] Generated new cluster ID: 741977263fd44c23a1580bc8e75a17f9
I20260619 16:48:25.121565 21861 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:48:25.151633 21861 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 16:48:25.153095 21861 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:48:25.164799 21861 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 04e20eba6005411cafece401b127948f: Generated new TSK 0
I20260619 16:48:25.166000 21861 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 16:48:25.177050 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu pid 21770
I20260619 16:48:25.178215 21832 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:48:25.270107 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu with pid 21770
W20260619 16:48:25.284080 21755 connection.cc:535] client connection to 127.0.0.1:38795 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:38795): Connection reset by peer (error 104)
2026-06-19T16:48:25Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1866 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781887385903558-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 16:48:25 dist-test-slave-wvz7 krb5kdc[21869](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 16:48:25 dist-test-slave-wvz7 krb5kdc[21869](info): set up 2 sockets
Jun 19 16:48:25 dist-test-slave-wvz7 krb5kdc[21869](info): commencing operation
krb5kdc: starting...
W20260619 16:48:26.289923 388 mini_kdc.cc:121] Time spent starting KDC: real 0.987s user 0.003s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781887385903558-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781887385903558-388-0/krb5kdc/test-user.keytab.
Jun 19 16:48:26 dist-test-slave-wvz7 krb5kdc[21869](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887706, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T16:48:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T16:48:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781887385903558-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781887385903558-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 16:48:26.525331 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu
/tmp/dist-test-taskEs2EH8/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781887385903558-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781887385903558-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781887385903558-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781887385903558-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33481
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781887385903558-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33091
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33481
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781887385903558-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781887385903558-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781887385903558-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:48:26.849804 21882 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:48:26.850147 21882 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:48:26.850286 21882 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:48:26.857601 21882 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:48:26.857759 21882 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:48:26.857842 21882 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:48:26.857923 21882 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:48:26.857980 21882 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:48:26.867816 21882 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33091
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781887385903558-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781887385903558-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33481
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781887385903558-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33481
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781887385903558-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781887385903558-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:48:26.869109 21882 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:48:26.870471 21882 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:48:26.881517 21888 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:48:26.881541 21887 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:48:26.881827 21890 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:48:26.882313 21882 server_base.cc:1034] running on GCE node
I20260619 16:48:26.883020 21882 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:48:26.884410 21882 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:48:26.885667 21882 hybrid_clock.cc:648] HybridClock initialized: now 1781887706885572 us; error 52 us; skew 500 ppm
I20260619 16:48:26.886056 21882 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:48:26 dist-test-slave-wvz7 krb5kdc[21869](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887706, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:48:26.892570 21882 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 16:48:26.894086 21882 webserver.cc:460] Webserver started at http://127.0.0.1:45049/ using document root <none> and password file <none>
I20260619 16:48:26.894774 21882 fs_manager.cc:359] Metadata directory not provided
I20260619 16:48:26.894873 21882 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:48:26.895123 21882 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:48:26.898113 21882 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781887385903558-388-0/minicluster-data/master-0/data/instance:
uuid: "9e3f7541868b491492cb18e3bb602bc2"
format_stamp: "Formatted at 2026-06-19 16:48:26 on dist-test-slave-wvz7"
server_key: "9e0cf8907f31db977c089ed328d1bae0"
server_key_iv: "732b49a1fb2eb108081cc87f0c2dee5c"
server_key_version: "encryptionkey@0"
I20260619 16:48:26.898811 21882 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781887385903558-388-0/minicluster-data/master-0/wal/instance:
uuid: "9e3f7541868b491492cb18e3bb602bc2"
format_stamp: "Formatted at 2026-06-19 16:48:26 on dist-test-slave-wvz7"
server_key: "9e0cf8907f31db977c089ed328d1bae0"
server_key_iv: "732b49a1fb2eb108081cc87f0c2dee5c"
server_key_version: "encryptionkey@0"
I20260619 16:48:26.903765 21882 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 16:48:26.907419 21897 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:48:26.908919 21882 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 16:48:26.909075 21882 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781887385903558-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781887385903558-388-0/minicluster-data/master-0/wal
uuid: "9e3f7541868b491492cb18e3bb602bc2"
format_stamp: "Formatted at 2026-06-19 16:48:26 on dist-test-slave-wvz7"
server_key: "9e0cf8907f31db977c089ed328d1bae0"
server_key_iv: "732b49a1fb2eb108081cc87f0c2dee5c"
server_key_version: "encryptionkey@0"
I20260619 16:48:26.909261 21882 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781887385903558-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781887385903558-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781887385903558-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:48:26.933921 21882 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:48:26.938373 21882 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:48:26.938657 21882 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:48:26.956032 21882 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33481
I20260619 16:48:26.956053 21948 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33481 every 8 connection(s)
I20260619 16:48:26.957392 21882 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781887385903558-388-0/minicluster-data/master-0/data/info.pb
I20260619 16:48:26.958781 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu as pid 21882
I20260619 16:48:26.958923 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781887385903558-388-0/minicluster-data/master-0/wal/instance
I20260619 16:48:26.959357 388 external_mini_cluster.cc:1428] Setting key b426d2ba551bf1bd5622b4f902fb90ca
I20260619 16:48:26.961956 21949 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:48:26.973419 21952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:48:26.961073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.135:39529 (local address 127.0.0.1:33481)
0619 16:48:26.961879 (+ 806us) server_negotiation.cc:206] Beginning negotiation
0619 16:48:26.961891 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 16:48:26.961933 (+ 42us) server_negotiation.cc:407] Connection header received
0619 16:48:26.963706 (+ 1773us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:48:26.963730 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:48:26.964160 (+ 430us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:48:26.964519 (+ 359us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:48:26.965240 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:48:26.966645 (+ 1405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:48:26.968667 (+ 2022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:48:26.969371 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:48:26.969504 (+ 133us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:48:26.970943 (+ 1439us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:48:26.970952 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:48:26.970968 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:48:26.971553 (+ 585us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:48:26.971664 (+ 111us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:48:26.971692 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:48:26.971829 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0619 16:48:26.972150 (+ 321us) server_negotiation.cc:299] Negotiation successful
0619 16:48:26.972353 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":572,"thread_start_us":334,"threads_started":1}
I20260619 16:48:26.975234 21949 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9e3f7541868b491492cb18e3bb602bc2: Bootstrap starting.
I20260619 16:48:26.977792 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 16:48:26.978806 21949 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9e3f7541868b491492cb18e3bb602bc2: Neither blocks nor log segments found. Creating new log.
I20260619 16:48:26.979863 21949 log.cc:824] T 00000000000000000000000000000000 P 9e3f7541868b491492cb18e3bb602bc2: Log is configured to *not* fsync() on all Append() calls
I20260619 16:48:26.983274 21949 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9e3f7541868b491492cb18e3bb602bc2: No bootstrap required, opened a new log
I20260619 16:48:26.988302 21952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:48:26.982131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.135:37879 (local address 127.0.0.1:33481)
0619 16:48:26.982276 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0619 16:48:26.982282 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:48:26.982710 (+ 428us) server_negotiation.cc:407] Connection header received
0619 16:48:26.982894 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:48:26.982900 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:48:26.982967 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:48:26.983050 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:48:26.983595 (+ 545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:48:26.984406 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:48:26.985774 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:48:26.986259 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:48:26.986377 (+ 118us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:48:26.987550 (+ 1173us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:48:26.987556 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:48:26.987559 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:48:26.987581 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:48:26.987628 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:48:26.987645 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:48:26.987762 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0619 16:48:26.987945 (+ 183us) server_negotiation.cc:299] Negotiation successful
0619 16:48:26.988002 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260619 16:48:26.991653 21949 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9e3f7541868b491492cb18e3bb602bc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e3f7541868b491492cb18e3bb602bc2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33481 } }
I20260619 16:48:26.992071 21949 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9e3f7541868b491492cb18e3bb602bc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:48:26.992164 21949 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9e3f7541868b491492cb18e3bb602bc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e3f7541868b491492cb18e3bb602bc2, State: Initialized, Role: FOLLOWER
I20260619 16:48:26.992789 21949 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9e3f7541868b491492cb18e3bb602bc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e3f7541868b491492cb18e3bb602bc2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33481 } }
I20260619 16:48:26.993005 21949 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9e3f7541868b491492cb18e3bb602bc2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:48:26.993129 21949 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9e3f7541868b491492cb18e3bb602bc2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:48:26.993294 21949 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9e3f7541868b491492cb18e3bb602bc2 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:48:26.995766 21949 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9e3f7541868b491492cb18e3bb602bc2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e3f7541868b491492cb18e3bb602bc2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33481 } }
I20260619 16:48:26.996363 21949 leader_election.cc:302] T 00000000000000000000000000000000 P 9e3f7541868b491492cb18e3bb602bc2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9e3f7541868b491492cb18e3bb602bc2; no voters:
I20260619 16:48:26.997041 21949 leader_election.cc:288] T 00000000000000000000000000000000 P 9e3f7541868b491492cb18e3bb602bc2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 16:48:26.997175 21959 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9e3f7541868b491492cb18e3bb602bc2 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:48:26.997848 21959 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9e3f7541868b491492cb18e3bb602bc2 [term 1 LEADER]: Becoming Leader. State: Replica: 9e3f7541868b491492cb18e3bb602bc2, State: Running, Role: LEADER
I20260619 16:48:26.998394 21959 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9e3f7541868b491492cb18e3bb602bc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e3f7541868b491492cb18e3bb602bc2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33481 } }
I20260619 16:48:26.999125 21949 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9e3f7541868b491492cb18e3bb602bc2 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:48:27.000890 21961 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9e3f7541868b491492cb18e3bb602bc2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9e3f7541868b491492cb18e3bb602bc2. Latest consensus state: current_term: 1 leader_uuid: "9e3f7541868b491492cb18e3bb602bc2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e3f7541868b491492cb18e3bb602bc2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33481 } } }
I20260619 16:48:27.001351 21961 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9e3f7541868b491492cb18e3bb602bc2 [sys.catalog]: This master's current role is: LEADER
I20260619 16:48:27.001492 21960 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9e3f7541868b491492cb18e3bb602bc2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9e3f7541868b491492cb18e3bb602bc2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e3f7541868b491492cb18e3bb602bc2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33481 } } }
I20260619 16:48:27.001930 21960 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9e3f7541868b491492cb18e3bb602bc2 [sys.catalog]: This master's current role is: LEADER
I20260619 16:48:27.003547 21964 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:48:27.008731 21964 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:48:27.017688 21964 catalog_manager.cc:1347] Generated new cluster ID: 61a636ba8ab64f74a1ef21b169466f1f
I20260619 16:48:27.017833 21964 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:48:27.056653 21964 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 16:48:27.058570 21964 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:48:27.070778 21964 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9e3f7541868b491492cb18e3bb602bc2: Generated new TSK 0
I20260619 16:48:27.071950 21964 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 16:48:27.090839 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu pid 21882
W20260619 16:48:27.092046 21944 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.135:39529
W20260619 16:48:27.092839 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu pid 21882: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 16:48:27.092979 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu with pid 21882
W20260619 16:48:27.106599 21867 connection.cc:535] client connection to 127.0.0.1:33481 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:33481): Connection reset by peer (error 104)
2026-06-19T16:48:27Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1822 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781887385903558-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 16:48:27 dist-test-slave-wvz7 krb5kdc[21980](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 16:48:27 dist-test-slave-wvz7 krb5kdc[21980](info): set up 2 sockets
Jun 19 16:48:27 dist-test-slave-wvz7 krb5kdc[21980](info): commencing operation
krb5kdc: starting...
W20260619 16:48:28.101948 388 mini_kdc.cc:121] Time spent starting KDC: real 0.977s user 0.001s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781887385903558-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781887385903558-388-0/krb5kdc/test-user.keytab.
Jun 19 16:48:28 dist-test-slave-wvz7 krb5kdc[21980](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887708, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T16:48:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T16:48:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781887385903558-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781887385903558-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 16:48:28.340483 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu
/tmp/dist-test-taskEs2EH8/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781887385903558-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781887385903558-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781887385903558-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781887385903558-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46819
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781887385903558-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44211
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46819
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781887385903558-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781887385903558-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781887385903558-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:48:28.655582 21993 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:48:28.655861 21993 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:48:28.655948 21993 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:48:28.662988 21993 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:48:28.663112 21993 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:48:28.663167 21993 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:48:28.663226 21993 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:48:28.663269 21993 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:48:28.673029 21993 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44211
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781887385903558-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781887385903558-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46819
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781887385903558-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46819
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781887385903558-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781887385903558-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:48:28.674319 21993 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:48:28.675536 21993 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:48:28.686676 21998 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:48:28.686730 21999 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:48:28.686944 22001 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:48:28.687397 21993 server_base.cc:1034] running on GCE node
I20260619 16:48:28.688112 21993 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:48:28.689467 21993 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:48:28.690701 21993 hybrid_clock.cc:648] HybridClock initialized: now 1781887708690638 us; error 42 us; skew 500 ppm
I20260619 16:48:28.691041 21993 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:48:28 dist-test-slave-wvz7 krb5kdc[21980](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887708, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:48:28.693929 21993 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 16:48:28.695271 21993 webserver.cc:460] Webserver started at http://127.0.0.1:42047/ using document root <none> and password file <none>
I20260619 16:48:28.695909 21993 fs_manager.cc:359] Metadata directory not provided
I20260619 16:48:28.696017 21993 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:48:28.696278 21993 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:48:28.699115 21993 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781887385903558-388-0/minicluster-data/master-0/data/instance:
uuid: "fa322ea0553348739c45900a85b38007"
format_stamp: "Formatted at 2026-06-19 16:48:28 on dist-test-slave-wvz7"
server_key: "d2876af895f3c4be05da8f7bc27eb356"
server_key_iv: "955254d7c2b9343df3bf13e0a0c22369"
server_key_version: "encryptionkey@0"
I20260619 16:48:28.699817 21993 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781887385903558-388-0/minicluster-data/master-0/wal/instance:
uuid: "fa322ea0553348739c45900a85b38007"
format_stamp: "Formatted at 2026-06-19 16:48:28 on dist-test-slave-wvz7"
server_key: "d2876af895f3c4be05da8f7bc27eb356"
server_key_iv: "955254d7c2b9343df3bf13e0a0c22369"
server_key_version: "encryptionkey@0"
I20260619 16:48:28.704537 21993 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260619 16:48:28.707659 22008 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:48:28.708869 21993 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 16:48:28.709009 21993 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781887385903558-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781887385903558-388-0/minicluster-data/master-0/wal
uuid: "fa322ea0553348739c45900a85b38007"
format_stamp: "Formatted at 2026-06-19 16:48:28 on dist-test-slave-wvz7"
server_key: "d2876af895f3c4be05da8f7bc27eb356"
server_key_iv: "955254d7c2b9343df3bf13e0a0c22369"
server_key_version: "encryptionkey@0"
I20260619 16:48:28.709167 21993 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781887385903558-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781887385903558-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781887385903558-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:48:28.714289 21993 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:48:28.717777 21993 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:48:28.718024 21993 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:48:28.733759 21993 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46819
I20260619 16:48:28.733812 22059 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46819 every 8 connection(s)
I20260619 16:48:28.735080 21993 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781887385903558-388-0/minicluster-data/master-0/data/info.pb
I20260619 16:48:28.738627 22060 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:48:28.743175 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu as pid 21993
I20260619 16:48:28.743319 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781887385903558-388-0/minicluster-data/master-0/wal/instance
I20260619 16:48:28.743709 388 external_mini_cluster.cc:1428] Setting key f8ad40d2bfd9ee942ff0a551e854997c
I20260619 16:48:28.748382 22060 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fa322ea0553348739c45900a85b38007: Bootstrap starting.
I20260619 16:48:28.752257 22060 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fa322ea0553348739c45900a85b38007: Neither blocks nor log segments found. Creating new log.
I20260619 16:48:28.753564 22060 log.cc:824] T 00000000000000000000000000000000 P fa322ea0553348739c45900a85b38007: Log is configured to *not* fsync() on all Append() calls
I20260619 16:48:28.756250 22063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:48:28.749474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.135:47125 (local address 127.0.0.1:46819)
0619 16:48:28.750337 (+ 863us) server_negotiation.cc:206] Beginning negotiation
0619 16:48:28.750352 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 16:48:28.750404 (+ 52us) server_negotiation.cc:407] Connection header received
0619 16:48:28.752755 (+ 2351us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:48:28.752783 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:48:28.753199 (+ 416us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:48:28.753655 (+ 456us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:48:28.754701 (+ 1046us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:48:28.754708 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:48:28.754720 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:48:28.754733 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:48:28.754787 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:48:28.754811 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:48:28.754954 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0619 16:48:28.755160 (+ 206us) server_negotiation.cc:299] Negotiation successful
0619 16:48:28.755343 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":596,"thread_start_us":359,"threads_started":1}
I20260619 16:48:28.756946 22060 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fa322ea0553348739c45900a85b38007: No bootstrap required, opened a new log
I20260619 16:48:28.759869 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 16:48:28.765149 22060 raft_consensus.cc:348] T 00000000000000000000000000000000 P fa322ea0553348739c45900a85b38007 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa322ea0553348739c45900a85b38007" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46819 } }
I20260619 16:48:28.765619 22060 raft_consensus.cc:374] T 00000000000000000000000000000000 P fa322ea0553348739c45900a85b38007 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:48:28.765748 22060 raft_consensus.cc:729] T 00000000000000000000000000000000 P fa322ea0553348739c45900a85b38007 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa322ea0553348739c45900a85b38007, State: Initialized, Role: FOLLOWER
I20260619 16:48:28.766369 22063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:48:28.763953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.135:41705 (local address 127.0.0.1:46819)
0619 16:48:28.764100 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0619 16:48:28.764104 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:48:28.764532 (+ 428us) server_negotiation.cc:407] Connection header received
0619 16:48:28.764713 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:48:28.764717 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:48:28.764768 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:48:28.764848 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:48:28.765768 (+ 920us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:48:28.765772 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:48:28.765777 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:48:28.765796 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:48:28.765842 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:48:28.765860 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:48:28.765967 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0619 16:48:28.766058 (+ 91us) server_negotiation.cc:299] Negotiation successful
0619 16:48:28.766123 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260619 16:48:28.766419 22060 consensus_queue.cc:260] T 00000000000000000000000000000000 P fa322ea0553348739c45900a85b38007 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa322ea0553348739c45900a85b38007" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46819 } }
I20260619 16:48:28.766727 22060 raft_consensus.cc:388] T 00000000000000000000000000000000 P fa322ea0553348739c45900a85b38007 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:48:28.766860 22060 raft_consensus.cc:482] T 00000000000000000000000000000000 P fa322ea0553348739c45900a85b38007 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:48:28.767024 22060 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fa322ea0553348739c45900a85b38007 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:48:28.769313 22060 raft_consensus.cc:504] T 00000000000000000000000000000000 P fa322ea0553348739c45900a85b38007 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa322ea0553348739c45900a85b38007" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46819 } }
I20260619 16:48:28.769850 22060 leader_election.cc:302] T 00000000000000000000000000000000 P fa322ea0553348739c45900a85b38007 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fa322ea0553348739c45900a85b38007; no voters:
I20260619 16:48:28.770391 22060 leader_election.cc:288] T 00000000000000000000000000000000 P fa322ea0553348739c45900a85b38007 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 16:48:28.770509 22070 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fa322ea0553348739c45900a85b38007 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:48:28.771121 22070 raft_consensus.cc:686] T 00000000000000000000000000000000 P fa322ea0553348739c45900a85b38007 [term 1 LEADER]: Becoming Leader. State: Replica: fa322ea0553348739c45900a85b38007, State: Running, Role: LEADER
I20260619 16:48:28.771610 22070 consensus_queue.cc:237] T 00000000000000000000000000000000 P fa322ea0553348739c45900a85b38007 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa322ea0553348739c45900a85b38007" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46819 } }
I20260619 16:48:28.772298 22060 sys_catalog.cc:564] T 00000000000000000000000000000000 P fa322ea0553348739c45900a85b38007 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:48:28.774422 22072 sys_catalog.cc:455] T 00000000000000000000000000000000 P fa322ea0553348739c45900a85b38007 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fa322ea0553348739c45900a85b38007. Latest consensus state: current_term: 1 leader_uuid: "fa322ea0553348739c45900a85b38007" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa322ea0553348739c45900a85b38007" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46819 } } }
I20260619 16:48:28.774842 22072 sys_catalog.cc:458] T 00000000000000000000000000000000 P fa322ea0553348739c45900a85b38007 [sys.catalog]: This master's current role is: LEADER
I20260619 16:48:28.775552 22071 sys_catalog.cc:455] T 00000000000000000000000000000000 P fa322ea0553348739c45900a85b38007 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fa322ea0553348739c45900a85b38007" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa322ea0553348739c45900a85b38007" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46819 } } }
I20260619 16:48:28.775833 22071 sys_catalog.cc:458] T 00000000000000000000000000000000 P fa322ea0553348739c45900a85b38007 [sys.catalog]: This master's current role is: LEADER
I20260619 16:48:28.778623 22075 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:48:28.784183 22075 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:48:28.792853 22075 catalog_manager.cc:1347] Generated new cluster ID: 32fa13d786ff4345b1b94ee5ef47f234
I20260619 16:48:28.792974 22075 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:48:28.810811 22075 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 16:48:28.811314 22075 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:48:28.825354 22075 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fa322ea0553348739c45900a85b38007: Generated new TSK 0
I20260619 16:48:28.826488 22075 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 16:48:28.834195 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu pid 21993
W20260619 16:48:28.835371 22055 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.135:47125
W20260619 16:48:28.836014 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu pid 21993: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 16:48:28.836176 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu with pid 21993
2026-06-19T16:48:28Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1743 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781887385903558-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 16:48:28 dist-test-slave-wvz7 krb5kdc[22091](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 16:48:28 dist-test-slave-wvz7 krb5kdc[22091](info): set up 2 sockets
Jun 19 16:48:28 dist-test-slave-wvz7 krb5kdc[22091](info): commencing operation
krb5kdc: starting...
W20260619 16:48:29.845867 388 mini_kdc.cc:121] Time spent starting KDC: real 0.978s user 0.007s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781887385903558-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781887385903558-388-0/krb5kdc/test-user.keytab.
Jun 19 16:48:29 dist-test-slave-wvz7 krb5kdc[22091](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887709, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T16:48:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T16:48:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781887385903558-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781887385903558-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260619 16:48:30.088672 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu
/tmp/dist-test-taskEs2EH8/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781887385903558-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781887385903558-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781887385903558-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781887385903558-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:44071
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781887385903558-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44827
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:44071
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781887385903558-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781887385903558-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781887385903558-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:48:30.405069 22104 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:48:30.405385 22104 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:48:30.405531 22104 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:48:30.413161 22104 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:48:30.413318 22104 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:48:30.413409 22104 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:48:30.413504 22104 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:48:30.413573 22104 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:48:30.423972 22104 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44827
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781887385903558-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781887385903558-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:44071
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781887385903558-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:44071
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781887385903558-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781887385903558-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:48:30.425374 22104 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:48:30.426766 22104 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:48:30.438203 22109 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:48:30.438271 22112 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:48:30.438248 22110 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:48:30.439353 22104 server_base.cc:1034] running on GCE node
I20260619 16:48:30.440110 22104 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:48:30.441519 22104 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:48:30.442742 22104 hybrid_clock.cc:648] HybridClock initialized: now 1781887710442676 us; error 55 us; skew 500 ppm
I20260619 16:48:30.443092 22104 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:48:30 dist-test-slave-wvz7 krb5kdc[22091](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887710, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:48:30.446040 22104 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260619 16:48:30.447432 22104 webserver.cc:460] Webserver started at http://127.0.97.62:37569/ using document root <none> and password file <none>
I20260619 16:48:30.448087 22104 fs_manager.cc:359] Metadata directory not provided
I20260619 16:48:30.448199 22104 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:48:30.448467 22104 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:48:30.451514 22104 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781887385903558-388-0/minicluster-data/master-0/data/instance:
uuid: "d56f9a79182e47cd90ce7c15a84a5c0c"
format_stamp: "Formatted at 2026-06-19 16:48:30 on dist-test-slave-wvz7"
server_key: "938a1e78b237f194a9b1bc3d456f35d5"
server_key_iv: "d8d62c5e86c239e71e165ccda1359460"
server_key_version: "encryptionkey@0"
I20260619 16:48:30.452255 22104 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781887385903558-388-0/minicluster-data/master-0/wal/instance:
uuid: "d56f9a79182e47cd90ce7c15a84a5c0c"
format_stamp: "Formatted at 2026-06-19 16:48:30 on dist-test-slave-wvz7"
server_key: "938a1e78b237f194a9b1bc3d456f35d5"
server_key_iv: "d8d62c5e86c239e71e165ccda1359460"
server_key_version: "encryptionkey@0"
I20260619 16:48:30.457263 22104 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260619 16:48:30.460937 22119 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:48:30.462396 22104 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 16:48:30.462548 22104 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781887385903558-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781887385903558-388-0/minicluster-data/master-0/wal
uuid: "d56f9a79182e47cd90ce7c15a84a5c0c"
format_stamp: "Formatted at 2026-06-19 16:48:30 on dist-test-slave-wvz7"
server_key: "938a1e78b237f194a9b1bc3d456f35d5"
server_key_iv: "d8d62c5e86c239e71e165ccda1359460"
server_key_version: "encryptionkey@0"
I20260619 16:48:30.462719 22104 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781887385903558-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781887385903558-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781887385903558-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:48:30.486510 22104 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:48:30.490700 22104 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:48:30.490963 22104 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:48:30.507980 22104 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:44071
I20260619 16:48:30.508018 22170 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:44071 every 8 connection(s)
I20260619 16:48:30.509285 22104 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781887385903558-388-0/minicluster-data/master-0/data/info.pb
I20260619 16:48:30.511606 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu as pid 22104
I20260619 16:48:30.511773 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781887385903558-388-0/minicluster-data/master-0/wal/instance
I20260619 16:48:30.512297 388 external_mini_cluster.cc:1428] Setting key b9a03452981ddbbe839b96176f451fff
I20260619 16:48:30.513377 22171 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Jun 19 16:48:30 dist-test-slave-wvz7 krb5kdc[22091](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887709, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 16:48:30.527096 22171 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d56f9a79182e47cd90ce7c15a84a5c0c: Bootstrap starting.
I20260619 16:48:30.530324 22171 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d56f9a79182e47cd90ce7c15a84a5c0c: Neither blocks nor log segments found. Creating new log.
I20260619 16:48:30.531490 22171 log.cc:824] T 00000000000000000000000000000000 P d56f9a79182e47cd90ce7c15a84a5c0c: Log is configured to *not* fsync() on all Append() calls
I20260619 16:48:30.534323 22174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:48:30.514023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36855 (local address 127.0.97.62:44071)
0619 16:48:30.514787 (+ 764us) server_negotiation.cc:206] Beginning negotiation
0619 16:48:30.514799 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 16:48:30.514837 (+ 38us) server_negotiation.cc:407] Connection header received
0619 16:48:30.516396 (+ 1559us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:48:30.516417 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:48:30.516824 (+ 407us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:48:30.517153 (+ 329us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:48:30.518781 (+ 1628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:48:30.520127 (+ 1346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:48:30.521870 (+ 1743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:48:30.522508 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:48:30.522672 (+ 164us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:48:30.527365 (+ 4693us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:48:30.527373 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:48:30.527388 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:48:30.527423 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:48:30.530314 (+ 2891us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:48:30.531128 (+ 814us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:48:30.531139 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:48:30.531145 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:48:30.531299 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:48:30.531764 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:48:30.531768 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:48:30.531770 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:48:30.532412 (+ 642us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:48:30.532769 (+ 357us) server_negotiation.cc:1035] Waiting for connection context
0619 16:48:30.533023 (+ 254us) server_negotiation.cc:299] Negotiation successful
0619 16:48:30.533299 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":577,"thread_start_us":306,"threads_started":1}
I20260619 16:48:30.534497 22171 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d56f9a79182e47cd90ce7c15a84a5c0c: No bootstrap required, opened a new log
I20260619 16:48:30.537998 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 16:48:30.543498 22171 raft_consensus.cc:348] T 00000000000000000000000000000000 P d56f9a79182e47cd90ce7c15a84a5c0c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d56f9a79182e47cd90ce7c15a84a5c0c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44071 } }
I20260619 16:48:30.543948 22171 raft_consensus.cc:374] T 00000000000000000000000000000000 P d56f9a79182e47cd90ce7c15a84a5c0c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:48:30.544061 22171 raft_consensus.cc:729] T 00000000000000000000000000000000 P d56f9a79182e47cd90ce7c15a84a5c0c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d56f9a79182e47cd90ce7c15a84a5c0c, State: Initialized, Role: FOLLOWER
I20260619 16:48:30.544731 22171 consensus_queue.cc:260] T 00000000000000000000000000000000 P d56f9a79182e47cd90ce7c15a84a5c0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d56f9a79182e47cd90ce7c15a84a5c0c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44071 } }
I20260619 16:48:30.544998 22171 raft_consensus.cc:388] T 00000000000000000000000000000000 P d56f9a79182e47cd90ce7c15a84a5c0c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:48:30.545125 22171 raft_consensus.cc:482] T 00000000000000000000000000000000 P d56f9a79182e47cd90ce7c15a84a5c0c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:48:30.545300 22171 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d56f9a79182e47cd90ce7c15a84a5c0c [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:48:30.548278 22171 raft_consensus.cc:504] T 00000000000000000000000000000000 P d56f9a79182e47cd90ce7c15a84a5c0c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d56f9a79182e47cd90ce7c15a84a5c0c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44071 } }
I20260619 16:48:30.548839 22171 leader_election.cc:302] T 00000000000000000000000000000000 P d56f9a79182e47cd90ce7c15a84a5c0c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d56f9a79182e47cd90ce7c15a84a5c0c; no voters:
I20260619 16:48:30.549458 22171 leader_election.cc:288] T 00000000000000000000000000000000 P d56f9a79182e47cd90ce7c15a84a5c0c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 16:48:30.549568 22181 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d56f9a79182e47cd90ce7c15a84a5c0c [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:48:30.551625 22171 sys_catalog.cc:564] T 00000000000000000000000000000000 P d56f9a79182e47cd90ce7c15a84a5c0c [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:48:30.552052 22181 raft_consensus.cc:686] T 00000000000000000000000000000000 P d56f9a79182e47cd90ce7c15a84a5c0c [term 1 LEADER]: Becoming Leader. State: Replica: d56f9a79182e47cd90ce7c15a84a5c0c, State: Running, Role: LEADER
I20260619 16:48:30.552675 22181 consensus_queue.cc:237] T 00000000000000000000000000000000 P d56f9a79182e47cd90ce7c15a84a5c0c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d56f9a79182e47cd90ce7c15a84a5c0c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44071 } }
I20260619 16:48:30.555713 22182 sys_catalog.cc:455] T 00000000000000000000000000000000 P d56f9a79182e47cd90ce7c15a84a5c0c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d56f9a79182e47cd90ce7c15a84a5c0c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d56f9a79182e47cd90ce7c15a84a5c0c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44071 } } }
I20260619 16:48:30.555852 22183 sys_catalog.cc:455] T 00000000000000000000000000000000 P d56f9a79182e47cd90ce7c15a84a5c0c [sys.catalog]: SysCatalogTable state changed. Reason: New leader d56f9a79182e47cd90ce7c15a84a5c0c. Latest consensus state: current_term: 1 leader_uuid: "d56f9a79182e47cd90ce7c15a84a5c0c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d56f9a79182e47cd90ce7c15a84a5c0c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44071 } } }
I20260619 16:48:30.556092 22182 sys_catalog.cc:458] T 00000000000000000000000000000000 P d56f9a79182e47cd90ce7c15a84a5c0c [sys.catalog]: This master's current role is: LEADER
I20260619 16:48:30.556110 22183 sys_catalog.cc:458] T 00000000000000000000000000000000 P d56f9a79182e47cd90ce7c15a84a5c0c [sys.catalog]: This master's current role is: LEADER
I20260619 16:48:30.556918 22190 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:48:30.560402 22174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:48:30.542999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47147 (local address 127.0.97.62:44071)
0619 16:48:30.543139 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0619 16:48:30.543143 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:48:30.543535 (+ 392us) server_negotiation.cc:407] Connection header received
0619 16:48:30.543687 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:48:30.543691 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:48:30.543751 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:48:30.543824 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:48:30.545500 (+ 1676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:48:30.546399 (+ 899us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:48:30.548203 (+ 1804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:48:30.548729 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:48:30.548822 (+ 93us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:48:30.551892 (+ 3070us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:48:30.551899 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:48:30.551905 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:48:30.551951 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:48:30.555974 (+ 4023us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:48:30.557118 (+ 1144us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:48:30.557125 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:48:30.557129 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:48:30.557318 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:48:30.557891 (+ 573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:48:30.557897 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:48:30.557901 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:48:30.558314 (+ 413us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:48:30.558670 (+ 356us) server_negotiation.cc:1035] Waiting for connection context
0619 16:48:30.559777 (+ 1107us) server_negotiation.cc:299] Negotiation successful
0619 16:48:30.559973 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260619 16:48:30.566553 22190 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:48:30.575628 22190 catalog_manager.cc:1347] Generated new cluster ID: 09e0d862e9a6452e822a3142c328d827
I20260619 16:48:30.575738 22190 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:48:30.590524 22190 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 16:48:30.591789 22190 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:48:30.600834 22190 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d56f9a79182e47cd90ce7c15a84a5c0c: Generated new TSK 0
I20260619 16:48:30.602010 22190 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 16:48:30.612840 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu pid 22104
I20260619 16:48:30.613921 22166 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:48:30.702623 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu with pid 22104
W20260619 16:48:30.716085 22089 connection.cc:535] client connection to 127.0.97.62:44071 recv error: Network error: failed to read from TLS socket (remote: 127.0.97.62:44071): Connection reset by peer (error 104)
2026-06-19T16:48:30Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1866 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781887385903558-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 16:48:30 dist-test-slave-wvz7 krb5kdc[22203](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 16:48:30 dist-test-slave-wvz7 krb5kdc[22203](info): set up 2 sockets
Jun 19 16:48:30 dist-test-slave-wvz7 krb5kdc[22203](info): commencing operation
krb5kdc: starting...
W20260619 16:48:31.718068 388 mini_kdc.cc:121] Time spent starting KDC: real 0.984s user 0.000s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781887385903558-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781887385903558-388-0/krb5kdc/test-user.keytab.
Jun 19 16:48:31 dist-test-slave-wvz7 krb5kdc[22203](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887711, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T16:48:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T16:48:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781887385903558-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781887385903558-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260619 16:48:31.938828 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu
/tmp/dist-test-taskEs2EH8/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781887385903558-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781887385903558-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781887385903558-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781887385903558-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37153
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781887385903558-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45519
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:37153
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781887385903558-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781887385903558-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781887385903558-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:48:32.250034 22216 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:48:32.250329 22216 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:48:32.250465 22216 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:48:32.257582 22216 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:48:32.257704 22216 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:48:32.257797 22216 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:48:32.257884 22216 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:48:32.257948 22216 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:48:32.267753 22216 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45519
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781887385903558-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781887385903558-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:37153
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781887385903558-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37153
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781887385903558-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781887385903558-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:48:32.269022 22216 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:48:32.270407 22216 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:48:32.281659 22221 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:48:32.281723 22222 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:48:32.281949 22224 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:48:32.282307 22216 server_base.cc:1034] running on GCE node
I20260619 16:48:32.283002 22216 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:48:32.284329 22216 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:48:32.285526 22216 hybrid_clock.cc:648] HybridClock initialized: now 1781887712285476 us; error 42 us; skew 500 ppm
I20260619 16:48:32.285887 22216 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:48:32 dist-test-slave-wvz7 krb5kdc[22203](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887712, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:48:32.291752 22216 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260619 16:48:32.293183 22216 webserver.cc:460] Webserver started at http://127.0.97.62:41737/ using document root <none> and password file <none>
I20260619 16:48:32.293889 22216 fs_manager.cc:359] Metadata directory not provided
I20260619 16:48:32.294020 22216 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:48:32.294327 22216 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:48:32.297289 22216 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781887385903558-388-0/minicluster-data/master-0/data/instance:
uuid: "791e73289e8242249624676f5b4e73c2"
format_stamp: "Formatted at 2026-06-19 16:48:32 on dist-test-slave-wvz7"
server_key: "297b041fea6824eee97c3058792c7bf2"
server_key_iv: "ffad7eaf0229a60968cbb8f33d49978d"
server_key_version: "encryptionkey@0"
I20260619 16:48:32.298100 22216 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781887385903558-388-0/minicluster-data/master-0/wal/instance:
uuid: "791e73289e8242249624676f5b4e73c2"
format_stamp: "Formatted at 2026-06-19 16:48:32 on dist-test-slave-wvz7"
server_key: "297b041fea6824eee97c3058792c7bf2"
server_key_iv: "ffad7eaf0229a60968cbb8f33d49978d"
server_key_version: "encryptionkey@0"
I20260619 16:48:32.303172 22216 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.002s
I20260619 16:48:32.306589 22231 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:48:32.307891 22216 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260619 16:48:32.308041 22216 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781887385903558-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781887385903558-388-0/minicluster-data/master-0/wal
uuid: "791e73289e8242249624676f5b4e73c2"
format_stamp: "Formatted at 2026-06-19 16:48:32 on dist-test-slave-wvz7"
server_key: "297b041fea6824eee97c3058792c7bf2"
server_key_iv: "ffad7eaf0229a60968cbb8f33d49978d"
server_key_version: "encryptionkey@0"
I20260619 16:48:32.308207 22216 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781887385903558-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781887385903558-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781887385903558-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:48:32.334128 22216 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:48:32.338320 22216 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:48:32.338583 22216 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:48:32.356048 22216 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:37153
I20260619 16:48:32.356065 22282 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:37153 every 8 connection(s)
I20260619 16:48:32.357321 22216 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781887385903558-388-0/minicluster-data/master-0/data/info.pb
I20260619 16:48:32.359764 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu as pid 22216
I20260619 16:48:32.359941 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781887385903558-388-0/minicluster-data/master-0/wal/instance
I20260619 16:48:32.360457 388 external_mini_cluster.cc:1428] Setting key 03512e35c0420ec4c3561a72530651d8
I20260619 16:48:32.361336 22283 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:48:32.372327 22286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:48:32.362234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39571 (local address 127.0.97.62:37153)
0619 16:48:32.362943 (+ 709us) server_negotiation.cc:206] Beginning negotiation
0619 16:48:32.362955 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 16:48:32.362992 (+ 37us) server_negotiation.cc:407] Connection header received
0619 16:48:32.364492 (+ 1500us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:48:32.364512 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:48:32.364886 (+ 374us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:48:32.365227 (+ 341us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:48:32.365758 (+ 531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:48:32.367076 (+ 1318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:48:32.368329 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:48:32.368973 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:48:32.369127 (+ 154us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:48:32.370280 (+ 1153us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:48:32.370287 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:48:32.370302 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:48:32.370833 (+ 531us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:48:32.370906 (+ 73us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:48:32.370930 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:48:32.371063 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0619 16:48:32.371308 (+ 245us) server_negotiation.cc:299] Negotiation successful
0619 16:48:32.371458 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":515,"thread_start_us":307,"threads_started":1}
I20260619 16:48:32.372756 22283 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 791e73289e8242249624676f5b4e73c2: Bootstrap starting.
I20260619 16:48:32.375748 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 16:48:32.375823 22283 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 791e73289e8242249624676f5b4e73c2: Neither blocks nor log segments found. Creating new log.
I20260619 16:48:32.376742 22283 log.cc:824] T 00000000000000000000000000000000 P 791e73289e8242249624676f5b4e73c2: Log is configured to *not* fsync() on all Append() calls
I20260619 16:48:32.379318 22283 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 791e73289e8242249624676f5b4e73c2: No bootstrap required, opened a new log
I20260619 16:48:32.386579 22286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:48:32.379978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47101 (local address 127.0.97.62:37153)
0619 16:48:32.380149 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0619 16:48:32.380155 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:48:32.380642 (+ 487us) server_negotiation.cc:407] Connection header received
0619 16:48:32.380826 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:48:32.380830 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:48:32.380890 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:48:32.380976 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:48:32.381515 (+ 539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:48:32.382339 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:48:32.384116 (+ 1777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:48:32.384624 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:48:32.384726 (+ 102us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:48:32.385868 (+ 1142us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:48:32.385873 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:48:32.385876 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:48:32.385892 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:48:32.385949 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:48:32.385966 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:48:32.386060 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0619 16:48:32.386218 (+ 158us) server_negotiation.cc:299] Negotiation successful
0619 16:48:32.386258 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260619 16:48:32.388314 22283 raft_consensus.cc:348] T 00000000000000000000000000000000 P 791e73289e8242249624676f5b4e73c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "791e73289e8242249624676f5b4e73c2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37153 } }
I20260619 16:48:32.388669 22283 raft_consensus.cc:374] T 00000000000000000000000000000000 P 791e73289e8242249624676f5b4e73c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:48:32.388757 22283 raft_consensus.cc:729] T 00000000000000000000000000000000 P 791e73289e8242249624676f5b4e73c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 791e73289e8242249624676f5b4e73c2, State: Initialized, Role: FOLLOWER
I20260619 16:48:32.389283 22283 consensus_queue.cc:260] T 00000000000000000000000000000000 P 791e73289e8242249624676f5b4e73c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "791e73289e8242249624676f5b4e73c2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37153 } }
I20260619 16:48:32.389494 22283 raft_consensus.cc:388] T 00000000000000000000000000000000 P 791e73289e8242249624676f5b4e73c2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:48:32.389612 22283 raft_consensus.cc:482] T 00000000000000000000000000000000 P 791e73289e8242249624676f5b4e73c2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:48:32.389789 22283 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 791e73289e8242249624676f5b4e73c2 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:48:32.391813 22283 raft_consensus.cc:504] T 00000000000000000000000000000000 P 791e73289e8242249624676f5b4e73c2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "791e73289e8242249624676f5b4e73c2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37153 } }
I20260619 16:48:32.392246 22283 leader_election.cc:302] T 00000000000000000000000000000000 P 791e73289e8242249624676f5b4e73c2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 791e73289e8242249624676f5b4e73c2; no voters:
I20260619 16:48:32.392760 22283 leader_election.cc:288] T 00000000000000000000000000000000 P 791e73289e8242249624676f5b4e73c2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 16:48:32.392900 22293 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 791e73289e8242249624676f5b4e73c2 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:48:32.393590 22293 raft_consensus.cc:686] T 00000000000000000000000000000000 P 791e73289e8242249624676f5b4e73c2 [term 1 LEADER]: Becoming Leader. State: Replica: 791e73289e8242249624676f5b4e73c2, State: Running, Role: LEADER
I20260619 16:48:32.394137 22293 consensus_queue.cc:237] T 00000000000000000000000000000000 P 791e73289e8242249624676f5b4e73c2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "791e73289e8242249624676f5b4e73c2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37153 } }
I20260619 16:48:32.394806 22283 sys_catalog.cc:564] T 00000000000000000000000000000000 P 791e73289e8242249624676f5b4e73c2 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:48:32.397066 22295 sys_catalog.cc:455] T 00000000000000000000000000000000 P 791e73289e8242249624676f5b4e73c2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 791e73289e8242249624676f5b4e73c2. Latest consensus state: current_term: 1 leader_uuid: "791e73289e8242249624676f5b4e73c2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "791e73289e8242249624676f5b4e73c2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37153 } } }
I20260619 16:48:32.397137 22294 sys_catalog.cc:455] T 00000000000000000000000000000000 P 791e73289e8242249624676f5b4e73c2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "791e73289e8242249624676f5b4e73c2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "791e73289e8242249624676f5b4e73c2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37153 } } }
I20260619 16:48:32.397410 22295 sys_catalog.cc:458] T 00000000000000000000000000000000 P 791e73289e8242249624676f5b4e73c2 [sys.catalog]: This master's current role is: LEADER
I20260619 16:48:32.397486 22294 sys_catalog.cc:458] T 00000000000000000000000000000000 P 791e73289e8242249624676f5b4e73c2 [sys.catalog]: This master's current role is: LEADER
I20260619 16:48:32.398219 22302 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:48:32.412927 22302 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:48:32.421651 22302 catalog_manager.cc:1347] Generated new cluster ID: d3e58c12bedd427dae79d6f8e14f3b10
I20260619 16:48:32.421784 22302 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:48:32.450423 22302 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 16:48:32.451687 22302 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:48:32.462160 22302 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 791e73289e8242249624676f5b4e73c2: Generated new TSK 0
I20260619 16:48:32.462985 22302 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 16:48:32.480211 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu pid 22216
W20260619 16:48:32.481266 22278 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:39571
W20260619 16:48:32.482054 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu pid 22216: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 16:48:32.482184 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu with pid 22216
W20260619 16:48:32.495676 22201 connection.cc:535] client connection to 127.0.97.62:37153 recv error: Network error: failed to read from TLS socket (remote: 127.0.97.62:37153): Connection reset by peer (error 104)
2026-06-19T16:48:32Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1779 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781887385903558-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 16:48:32 dist-test-slave-wvz7 krb5kdc[22314](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 16:48:32 dist-test-slave-wvz7 krb5kdc[22314](info): set up 2 sockets
Jun 19 16:48:32 dist-test-slave-wvz7 krb5kdc[22314](info): commencing operation
krb5kdc: starting...
W20260619 16:48:33.501968 388 mini_kdc.cc:121] Time spent starting KDC: real 0.988s user 0.002s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781887385903558-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781887385903558-388-0/krb5kdc/test-user.keytab.
Jun 19 16:48:33 dist-test-slave-wvz7 krb5kdc[22314](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887713, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T16:48:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T16:48:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781887385903558-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781887385903558-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260619 16:48:33.723029 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu
/tmp/dist-test-taskEs2EH8/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781887385903558-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781887385903558-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781887385903558-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781887385903558-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45209
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781887385903558-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45319
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:45209
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781887385903558-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781887385903558-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781887385903558-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:48:34.032759 22327 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:48:34.033051 22327 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:48:34.033149 22327 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:48:34.040323 22327 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:48:34.040431 22327 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:48:34.040482 22327 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:48:34.040534 22327 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:48:34.040575 22327 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:48:34.050172 22327 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45319
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781887385903558-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781887385903558-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:45209
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781887385903558-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45209
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781887385903558-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781887385903558-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:48:34.051306 22327 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:48:34.052490 22327 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:48:34.063300 22335 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:48:34.063369 22332 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:48:34.063395 22333 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:48:34.064013 22327 server_base.cc:1034] running on GCE node
I20260619 16:48:34.064702 22327 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:48:34.066043 22327 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:48:34.067227 22327 hybrid_clock.cc:648] HybridClock initialized: now 1781887714067178 us; error 44 us; skew 500 ppm
I20260619 16:48:34.067536 22327 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:48:34 dist-test-slave-wvz7 krb5kdc[22314](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887714, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:48:34.070454 22327 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260619 16:48:34.071749 22327 webserver.cc:460] Webserver started at http://127.0.97.62:42671/ using document root <none> and password file <none>
I20260619 16:48:34.072355 22327 fs_manager.cc:359] Metadata directory not provided
I20260619 16:48:34.072450 22327 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:48:34.072701 22327 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:48:34.075551 22327 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781887385903558-388-0/minicluster-data/master-0/data/instance:
uuid: "2c2f9bc010424cd98940328851301d04"
format_stamp: "Formatted at 2026-06-19 16:48:34 on dist-test-slave-wvz7"
server_key: "8a78618baeda85d1ebe10dd10ac2e111"
server_key_iv: "1623f488266ab54ba14ed43dfa813ccd"
server_key_version: "encryptionkey@0"
I20260619 16:48:34.076237 22327 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781887385903558-388-0/minicluster-data/master-0/wal/instance:
uuid: "2c2f9bc010424cd98940328851301d04"
format_stamp: "Formatted at 2026-06-19 16:48:34 on dist-test-slave-wvz7"
server_key: "8a78618baeda85d1ebe10dd10ac2e111"
server_key_iv: "1623f488266ab54ba14ed43dfa813ccd"
server_key_version: "encryptionkey@0"
I20260619 16:48:34.081038 22327 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260619 16:48:34.084316 22342 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:48:34.085604 22327 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260619 16:48:34.085773 22327 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781887385903558-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781887385903558-388-0/minicluster-data/master-0/wal
uuid: "2c2f9bc010424cd98940328851301d04"
format_stamp: "Formatted at 2026-06-19 16:48:34 on dist-test-slave-wvz7"
server_key: "8a78618baeda85d1ebe10dd10ac2e111"
server_key_iv: "1623f488266ab54ba14ed43dfa813ccd"
server_key_version: "encryptionkey@0"
I20260619 16:48:34.085954 22327 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781887385903558-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781887385903558-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781887385903558-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:48:34.091114 22327 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:48:34.094975 22327 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:48:34.095204 22327 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:48:34.111441 22327 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:45209
I20260619 16:48:34.111508 22393 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:45209 every 8 connection(s)
I20260619 16:48:34.112778 22327 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781887385903558-388-0/minicluster-data/master-0/data/info.pb
I20260619 16:48:34.113480 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu as pid 22327
I20260619 16:48:34.113636 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781887385903558-388-0/minicluster-data/master-0/wal/instance
I20260619 16:48:34.114197 388 external_mini_cluster.cc:1428] Setting key a0524ba184f0affbc1cb27fb20e8cb3b
I20260619 16:48:34.117017 22394 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:48:34.121549 22397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:48:34.115965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48639 (local address 127.0.97.62:45209)
0619 16:48:34.116717 (+ 752us) server_negotiation.cc:206] Beginning negotiation
0619 16:48:34.116729 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 16:48:34.116769 (+ 40us) server_negotiation.cc:407] Connection header received
0619 16:48:34.118363 (+ 1594us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:48:34.118385 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:48:34.118715 (+ 330us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:48:34.119043 (+ 328us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:48:34.120015 (+ 972us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:48:34.120020 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:48:34.120032 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:48:34.120047 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:48:34.120121 (+ 74us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:48:34.120144 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:48:34.120346 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0619 16:48:34.120491 (+ 145us) server_negotiation.cc:299] Negotiation successful
0619 16:48:34.120651 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":511,"thread_start_us":283,"threads_started":1}
I20260619 16:48:34.125444 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 16:48:34.128302 22394 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2c2f9bc010424cd98940328851301d04: Bootstrap starting.
I20260619 16:48:34.131453 22394 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2c2f9bc010424cd98940328851301d04: Neither blocks nor log segments found. Creating new log.
I20260619 16:48:34.132005 22397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:48:34.129535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48885 (local address 127.0.97.62:45209)
0619 16:48:34.129757 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0619 16:48:34.129764 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:48:34.130190 (+ 426us) server_negotiation.cc:407] Connection header received
0619 16:48:34.130349 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:48:34.130354 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:48:34.130403 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:48:34.130480 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:48:34.131386 (+ 906us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:48:34.131391 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:48:34.131394 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:48:34.131409 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:48:34.131454 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:48:34.131472 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:48:34.131573 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0619 16:48:34.131676 (+ 103us) server_negotiation.cc:299] Negotiation successful
0619 16:48:34.131743 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260619 16:48:34.132611 22394 log.cc:824] T 00000000000000000000000000000000 P 2c2f9bc010424cd98940328851301d04: Log is configured to *not* fsync() on all Append() calls
I20260619 16:48:34.135277 22394 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2c2f9bc010424cd98940328851301d04: No bootstrap required, opened a new log
I20260619 16:48:34.142561 22394 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2c2f9bc010424cd98940328851301d04 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c2f9bc010424cd98940328851301d04" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45209 } }
I20260619 16:48:34.142902 22394 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2c2f9bc010424cd98940328851301d04 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:48:34.142972 22394 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2c2f9bc010424cd98940328851301d04 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c2f9bc010424cd98940328851301d04, State: Initialized, Role: FOLLOWER
I20260619 16:48:34.143455 22394 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2c2f9bc010424cd98940328851301d04 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c2f9bc010424cd98940328851301d04" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45209 } }
I20260619 16:48:34.143635 22394 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2c2f9bc010424cd98940328851301d04 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:48:34.143725 22394 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2c2f9bc010424cd98940328851301d04 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:48:34.143833 22394 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2c2f9bc010424cd98940328851301d04 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:48:34.145932 22394 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2c2f9bc010424cd98940328851301d04 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c2f9bc010424cd98940328851301d04" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45209 } }
I20260619 16:48:34.146399 22394 leader_election.cc:302] T 00000000000000000000000000000000 P 2c2f9bc010424cd98940328851301d04 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2c2f9bc010424cd98940328851301d04; no voters:
I20260619 16:48:34.147048 22394 leader_election.cc:288] T 00000000000000000000000000000000 P 2c2f9bc010424cd98940328851301d04 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 16:48:34.147153 22404 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2c2f9bc010424cd98940328851301d04 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:48:34.147722 22404 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2c2f9bc010424cd98940328851301d04 [term 1 LEADER]: Becoming Leader. State: Replica: 2c2f9bc010424cd98940328851301d04, State: Running, Role: LEADER
I20260619 16:48:34.148313 22404 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2c2f9bc010424cd98940328851301d04 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c2f9bc010424cd98940328851301d04" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45209 } }
I20260619 16:48:34.149173 22394 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2c2f9bc010424cd98940328851301d04 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:48:34.151165 22406 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c2f9bc010424cd98940328851301d04 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2c2f9bc010424cd98940328851301d04. Latest consensus state: current_term: 1 leader_uuid: "2c2f9bc010424cd98940328851301d04" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c2f9bc010424cd98940328851301d04" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45209 } } }
I20260619 16:48:34.151580 22406 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c2f9bc010424cd98940328851301d04 [sys.catalog]: This master's current role is: LEADER
I20260619 16:48:34.151194 22405 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c2f9bc010424cd98940328851301d04 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2c2f9bc010424cd98940328851301d04" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c2f9bc010424cd98940328851301d04" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45209 } } }
I20260619 16:48:34.152070 22405 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c2f9bc010424cd98940328851301d04 [sys.catalog]: This master's current role is: LEADER
I20260619 16:48:34.153754 22412 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:48:34.158989 22412 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:48:34.168633 22412 catalog_manager.cc:1347] Generated new cluster ID: 00ff02a383a8453d93bec90ba5d96d67
I20260619 16:48:34.168741 22412 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:48:34.181610 22412 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 16:48:34.182184 22412 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:48:34.198609 22412 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2c2f9bc010424cd98940328851301d04: Generated new TSK 0
I20260619 16:48:34.199473 22412 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 16:48:34.205080 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu pid 22327
W20260619 16:48:34.206713 22389 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:48639
W20260619 16:48:34.207311 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu pid 22327: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 16:48:34.207437 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu with pid 22327
2026-06-19T16:48:34Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1725 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781887385903558-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 16:48:34 dist-test-slave-wvz7 krb5kdc[22425](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 16:48:34 dist-test-slave-wvz7 krb5kdc[22425](info): set up 2 sockets
Jun 19 16:48:34 dist-test-slave-wvz7 krb5kdc[22425](info): commencing operation
krb5kdc: starting...
W20260619 16:48:35.213706 388 mini_kdc.cc:121] Time spent starting KDC: real 0.975s user 0.005s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781887385903558-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781887385903558-388-0/krb5kdc/test-user.keytab.
Jun 19 16:48:35 dist-test-slave-wvz7 krb5kdc[22425](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887715, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T16:48:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T16:48:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781887385903558-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781887385903558-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260619 16:48:35.432515 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu
/tmp/dist-test-taskEs2EH8/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781887385903558-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781887385903558-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781887385903558-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781887385903558-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:43879
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781887385903558-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43485
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:43879
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781887385903558-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781887385903558-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781887385903558-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:48:35.746016 22438 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:48:35.746292 22438 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:48:35.746410 22438 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:48:35.753870 22438 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:48:35.753968 22438 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:48:35.754060 22438 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:48:35.754140 22438 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:48:35.754199 22438 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:48:35.764019 22438 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43485
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781887385903558-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781887385903558-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:43879
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781887385903558-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:43879
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781887385903558-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781887385903558-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:48:35.765295 22438 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:48:35.766588 22438 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:48:35.777577 22446 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:48:35.777616 22444 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:48:35.777642 22443 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:48:35.778668 22438 server_base.cc:1034] running on GCE node
I20260619 16:48:35.779656 22438 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:48:35.781474 22438 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:48:35.782855 22438 hybrid_clock.cc:648] HybridClock initialized: now 1781887715782805 us; error 30 us; skew 500 ppm
I20260619 16:48:35.783279 22438 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:48:35 dist-test-slave-wvz7 krb5kdc[22425](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887715, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:48:35.787098 22438 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260619 16:48:35.788399 22438 webserver.cc:460] Webserver started at http://127.0.97.62:45075/ using document root <none> and password file <none>
I20260619 16:48:35.788995 22438 fs_manager.cc:359] Metadata directory not provided
I20260619 16:48:35.789090 22438 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:48:35.789320 22438 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:48:35.792207 22438 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781887385903558-388-0/minicluster-data/master-0/data/instance:
uuid: "e306d205601840b4b5de11a8463960b8"
format_stamp: "Formatted at 2026-06-19 16:48:35 on dist-test-slave-wvz7"
server_key: "c6f2554b416ff2fdd36537bae83644a2"
server_key_iv: "5a605388f4344bf611b99b0524b41cb3"
server_key_version: "encryptionkey@0"
I20260619 16:48:35.792886 22438 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781887385903558-388-0/minicluster-data/master-0/wal/instance:
uuid: "e306d205601840b4b5de11a8463960b8"
format_stamp: "Formatted at 2026-06-19 16:48:35 on dist-test-slave-wvz7"
server_key: "c6f2554b416ff2fdd36537bae83644a2"
server_key_iv: "5a605388f4344bf611b99b0524b41cb3"
server_key_version: "encryptionkey@0"
I20260619 16:48:35.797763 22438 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.002s
I20260619 16:48:35.801146 22453 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:48:35.802417 22438 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 16:48:35.802553 22438 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781887385903558-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781887385903558-388-0/minicluster-data/master-0/wal
uuid: "e306d205601840b4b5de11a8463960b8"
format_stamp: "Formatted at 2026-06-19 16:48:35 on dist-test-slave-wvz7"
server_key: "c6f2554b416ff2fdd36537bae83644a2"
server_key_iv: "5a605388f4344bf611b99b0524b41cb3"
server_key_version: "encryptionkey@0"
I20260619 16:48:35.802734 22438 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781887385903558-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781887385903558-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781887385903558-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:48:35.817024 22438 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:48:35.820729 22438 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:48:35.820959 22438 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:48:35.836756 22438 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:43879
I20260619 16:48:35.836763 22504 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:43879 every 8 connection(s)
I20260619 16:48:35.838080 22438 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781887385903558-388-0/minicluster-data/master-0/data/info.pb
I20260619 16:48:35.841653 22505 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:48:35.842139 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu as pid 22438
I20260619 16:48:35.842283 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781887385903558-388-0/minicluster-data/master-0/wal/instance
I20260619 16:48:35.842700 388 external_mini_cluster.cc:1428] Setting key ecd87f616b45d8d7f94f1d90c21c6e88
I20260619 16:48:35.854231 22505 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e306d205601840b4b5de11a8463960b8: Bootstrap starting.
Jun 19 16:48:35 dist-test-slave-wvz7 krb5kdc[22425](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887715, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 16:48:35.857609 22505 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e306d205601840b4b5de11a8463960b8: Neither blocks nor log segments found. Creating new log.
I20260619 16:48:35.858640 22505 log.cc:824] T 00000000000000000000000000000000 P e306d205601840b4b5de11a8463960b8: Log is configured to *not* fsync() on all Append() calls
I20260619 16:48:35.861218 22505 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e306d205601840b4b5de11a8463960b8: No bootstrap required, opened a new log
I20260619 16:48:35.864431 22508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:48:35.844276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42799 (local address 127.0.97.62:43879)
0619 16:48:35.844998 (+ 722us) server_negotiation.cc:206] Beginning negotiation
0619 16:48:35.845008 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 16:48:35.845053 (+ 45us) server_negotiation.cc:407] Connection header received
0619 16:48:35.847191 (+ 2138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:48:35.847212 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:48:35.847600 (+ 388us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:48:35.847905 (+ 305us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:48:35.849493 (+ 1588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:48:35.850808 (+ 1315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:48:35.852609 (+ 1801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:48:35.853231 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:48:35.853387 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:48:35.857854 (+ 4467us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:48:35.857864 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:48:35.857878 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:48:35.857917 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:48:35.860725 (+ 2808us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:48:35.861342 (+ 617us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:48:35.861349 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:48:35.861354 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:48:35.861515 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:48:35.861902 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:48:35.861905 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:48:35.861908 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:48:35.862385 (+ 477us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:48:35.862667 (+ 282us) server_negotiation.cc:1035] Waiting for connection context
0619 16:48:35.863104 (+ 437us) server_negotiation.cc:299] Negotiation successful
0619 16:48:35.863483 (+ 379us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":534,"thread_start_us":322,"threads_started":1}
I20260619 16:48:35.867830 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 16:48:35.868785 22505 raft_consensus.cc:348] T 00000000000000000000000000000000 P e306d205601840b4b5de11a8463960b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e306d205601840b4b5de11a8463960b8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43879 } }
I20260619 16:48:35.869103 22505 raft_consensus.cc:374] T 00000000000000000000000000000000 P e306d205601840b4b5de11a8463960b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:48:35.869184 22505 raft_consensus.cc:729] T 00000000000000000000000000000000 P e306d205601840b4b5de11a8463960b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e306d205601840b4b5de11a8463960b8, State: Initialized, Role: FOLLOWER
I20260619 16:48:35.869683 22505 consensus_queue.cc:260] T 00000000000000000000000000000000 P e306d205601840b4b5de11a8463960b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e306d205601840b4b5de11a8463960b8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43879 } }
I20260619 16:48:35.869936 22505 raft_consensus.cc:388] T 00000000000000000000000000000000 P e306d205601840b4b5de11a8463960b8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:48:35.870049 22505 raft_consensus.cc:482] T 00000000000000000000000000000000 P e306d205601840b4b5de11a8463960b8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:48:35.870183 22505 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e306d205601840b4b5de11a8463960b8 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:48:35.872539 22505 raft_consensus.cc:504] T 00000000000000000000000000000000 P e306d205601840b4b5de11a8463960b8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e306d205601840b4b5de11a8463960b8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43879 } }
I20260619 16:48:35.873111 22505 leader_election.cc:302] T 00000000000000000000000000000000 P e306d205601840b4b5de11a8463960b8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e306d205601840b4b5de11a8463960b8; no voters:
I20260619 16:48:35.873703 22505 leader_election.cc:288] T 00000000000000000000000000000000 P e306d205601840b4b5de11a8463960b8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 16:48:35.873821 22515 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e306d205601840b4b5de11a8463960b8 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:48:35.874959 22515 raft_consensus.cc:686] T 00000000000000000000000000000000 P e306d205601840b4b5de11a8463960b8 [term 1 LEADER]: Becoming Leader. State: Replica: e306d205601840b4b5de11a8463960b8, State: Running, Role: LEADER
I20260619 16:48:35.875478 22515 consensus_queue.cc:237] T 00000000000000000000000000000000 P e306d205601840b4b5de11a8463960b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e306d205601840b4b5de11a8463960b8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43879 } }
I20260619 16:48:35.875837 22505 sys_catalog.cc:564] T 00000000000000000000000000000000 P e306d205601840b4b5de11a8463960b8 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:48:35.878010 22517 sys_catalog.cc:455] T 00000000000000000000000000000000 P e306d205601840b4b5de11a8463960b8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e306d205601840b4b5de11a8463960b8. Latest consensus state: current_term: 1 leader_uuid: "e306d205601840b4b5de11a8463960b8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e306d205601840b4b5de11a8463960b8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43879 } } }
I20260619 16:48:35.878402 22516 sys_catalog.cc:455] T 00000000000000000000000000000000 P e306d205601840b4b5de11a8463960b8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e306d205601840b4b5de11a8463960b8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e306d205601840b4b5de11a8463960b8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43879 } } }
I20260619 16:48:35.878980 22517 sys_catalog.cc:458] T 00000000000000000000000000000000 P e306d205601840b4b5de11a8463960b8 [sys.catalog]: This master's current role is: LEADER
I20260619 16:48:35.879509 22516 sys_catalog.cc:458] T 00000000000000000000000000000000 P e306d205601840b4b5de11a8463960b8 [sys.catalog]: This master's current role is: LEADER
I20260619 16:48:35.884826 22524 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:48:35.890231 22508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:48:35.871889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38985 (local address 127.0.97.62:43879)
0619 16:48:35.872053 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0619 16:48:35.872060 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:48:35.872472 (+ 412us) server_negotiation.cc:407] Connection header received
0619 16:48:35.872595 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:48:35.872600 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:48:35.872675 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:48:35.872767 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:48:35.874700 (+ 1933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:48:35.875782 (+ 1082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:48:35.877580 (+ 1798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:48:35.878416 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:48:35.878561 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:48:35.881623 (+ 3062us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:48:35.881630 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:48:35.881636 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:48:35.881675 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:48:35.886089 (+ 4414us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:48:35.887024 (+ 935us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:48:35.887029 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:48:35.887034 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:48:35.887185 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:48:35.887610 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:48:35.887617 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:48:35.887620 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:48:35.888067 (+ 447us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:48:35.889448 (+ 1381us) server_negotiation.cc:1035] Waiting for connection context
0619 16:48:35.889596 (+ 148us) server_negotiation.cc:299] Negotiation successful
0619 16:48:35.889843 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260619 16:48:35.897193 22524 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:48:35.912020 22524 catalog_manager.cc:1347] Generated new cluster ID: db8def2c3a0045128f2e9885434bf92b
I20260619 16:48:35.912140 22524 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:48:35.930509 22524 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 16:48:35.931854 22524 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:48:35.943608 22524 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e306d205601840b4b5de11a8463960b8: Generated new TSK 0
I20260619 16:48:35.944456 22524 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 16:48:35.955837 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu pid 22438
I20260619 16:48:35.956863 22500 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:48:36.047432 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu with pid 22438
2026-06-19T16:48:36Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1839 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781887385903558-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 16:48:36 dist-test-slave-wvz7 krb5kdc[22537](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 16:48:36 dist-test-slave-wvz7 krb5kdc[22537](info): set up 2 sockets
Jun 19 16:48:36 dist-test-slave-wvz7 krb5kdc[22537](info): commencing operation
krb5kdc: starting...
W20260619 16:48:37.046065 388 mini_kdc.cc:121] Time spent starting KDC: real 0.967s user 0.001s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781887385903558-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781887385903558-388-0/krb5kdc/test-user.keytab.
Jun 19 16:48:37 dist-test-slave-wvz7 krb5kdc[22537](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887717, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T16:48:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T16:48:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781887385903558-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781887385903558-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260619 16:48:37.276701 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu
/tmp/dist-test-taskEs2EH8/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781887385903558-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781887385903558-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781887385903558-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781887385903558-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37549
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781887385903558-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36553
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:37549
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781887385903558-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781887385903558-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781887385903558-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:48:37.596274 22550 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:48:37.596550 22550 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:48:37.596637 22550 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:48:37.603590 22550 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:48:37.603685 22550 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:48:37.603739 22550 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:48:37.603798 22550 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:48:37.603840 22550 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:48:37.613241 22550 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36553
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781887385903558-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781887385903558-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:37549
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781887385903558-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37549
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781887385903558-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781887385903558-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:48:37.614457 22550 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:48:37.615646 22550 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:48:37.626199 22555 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:48:37.626277 22558 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:48:37.626250 22556 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:48:37.627013 22550 server_base.cc:1034] running on GCE node
I20260619 16:48:37.627718 22550 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:48:37.629016 22550 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:48:37.630219 22550 hybrid_clock.cc:648] HybridClock initialized: now 1781887717630177 us; error 33 us; skew 500 ppm
I20260619 16:48:37.630575 22550 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:48:37 dist-test-slave-wvz7 krb5kdc[22537](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887717, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:48:37.636435 22550 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260619 16:48:37.637933 22550 webserver.cc:460] Webserver started at http://127.0.97.62:41337/ using document root <none> and password file <none>
I20260619 16:48:37.638577 22550 fs_manager.cc:359] Metadata directory not provided
I20260619 16:48:37.638687 22550 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:48:37.638954 22550 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:48:37.641785 22550 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781887385903558-388-0/minicluster-data/master-0/data/instance:
uuid: "f6b682a6ffac4d2789a278e5a2bb5eb1"
format_stamp: "Formatted at 2026-06-19 16:48:37 on dist-test-slave-wvz7"
server_key: "2c7baa7a74adbba0a14bac43760658be"
server_key_iv: "ce64c12a23c8a8fa4c3884ffda547744"
server_key_version: "encryptionkey@0"
I20260619 16:48:37.642503 22550 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781887385903558-388-0/minicluster-data/master-0/wal/instance:
uuid: "f6b682a6ffac4d2789a278e5a2bb5eb1"
format_stamp: "Formatted at 2026-06-19 16:48:37 on dist-test-slave-wvz7"
server_key: "2c7baa7a74adbba0a14bac43760658be"
server_key_iv: "ce64c12a23c8a8fa4c3884ffda547744"
server_key_version: "encryptionkey@0"
I20260619 16:48:37.647348 22550 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260619 16:48:37.650656 22565 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:48:37.652014 22550 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 16:48:37.652160 22550 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781887385903558-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781887385903558-388-0/minicluster-data/master-0/wal
uuid: "f6b682a6ffac4d2789a278e5a2bb5eb1"
format_stamp: "Formatted at 2026-06-19 16:48:37 on dist-test-slave-wvz7"
server_key: "2c7baa7a74adbba0a14bac43760658be"
server_key_iv: "ce64c12a23c8a8fa4c3884ffda547744"
server_key_version: "encryptionkey@0"
I20260619 16:48:37.652346 22550 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781887385903558-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781887385903558-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781887385903558-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:48:37.678103 22550 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:48:37.682034 22550 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:48:37.682281 22550 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:48:37.698779 22550 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:37549
I20260619 16:48:37.698810 22616 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:37549 every 8 connection(s)
I20260619 16:48:37.700042 22550 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781887385903558-388-0/minicluster-data/master-0/data/info.pb
I20260619 16:48:37.701392 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu as pid 22550
I20260619 16:48:37.701567 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781887385903558-388-0/minicluster-data/master-0/wal/instance
I20260619 16:48:37.702152 388 external_mini_cluster.cc:1428] Setting key 065180505e87918a8b6186695c2c7294
I20260619 16:48:37.704293 22617 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:48:37.714444 22620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:48:37.703801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34927 (local address 127.0.97.62:37549)
0619 16:48:37.704507 (+ 706us) server_negotiation.cc:206] Beginning negotiation
0619 16:48:37.704518 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 16:48:37.704561 (+ 43us) server_negotiation.cc:407] Connection header received
0619 16:48:37.706111 (+ 1550us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:48:37.706130 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:48:37.706510 (+ 380us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:48:37.706822 (+ 312us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:48:37.707365 (+ 543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:48:37.708628 (+ 1263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:48:37.710386 (+ 1758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:48:37.711008 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:48:37.711122 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:48:37.712305 (+ 1183us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:48:37.712316 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:48:37.712328 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:48:37.712858 (+ 530us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:48:37.712928 (+ 70us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:48:37.712951 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:48:37.713032 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0619 16:48:37.713240 (+ 208us) server_negotiation.cc:299] Negotiation successful
0619 16:48:37.713508 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":521,"thread_start_us":287,"threads_started":1}
I20260619 16:48:37.715682 22617 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f6b682a6ffac4d2789a278e5a2bb5eb1: Bootstrap starting.
I20260619 16:48:37.717723 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 16:48:37.718619 22617 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f6b682a6ffac4d2789a278e5a2bb5eb1: Neither blocks nor log segments found. Creating new log.
I20260619 16:48:37.719499 22617 log.cc:824] T 00000000000000000000000000000000 P f6b682a6ffac4d2789a278e5a2bb5eb1: Log is configured to *not* fsync() on all Append() calls
I20260619 16:48:37.722019 22617 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f6b682a6ffac4d2789a278e5a2bb5eb1: No bootstrap required, opened a new log
I20260619 16:48:37.727514 22620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:48:37.721629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40829 (local address 127.0.97.62:37549)
0619 16:48:37.721826 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0619 16:48:37.721831 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:48:37.722162 (+ 331us) server_negotiation.cc:407] Connection header received
0619 16:48:37.722293 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:48:37.722296 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:48:37.722347 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:48:37.722416 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:48:37.722928 (+ 512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:48:37.723596 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:48:37.725370 (+ 1774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:48:37.725851 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:48:37.725941 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:48:37.726909 (+ 968us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:48:37.726914 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:48:37.726917 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:48:37.726935 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:48:37.726972 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:48:37.726986 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:48:37.727053 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0619 16:48:37.727180 (+ 127us) server_negotiation.cc:299] Negotiation successful
0619 16:48:37.727243 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260619 16:48:37.730490 22617 raft_consensus.cc:348] T 00000000000000000000000000000000 P f6b682a6ffac4d2789a278e5a2bb5eb1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6b682a6ffac4d2789a278e5a2bb5eb1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37549 } }
I20260619 16:48:37.730840 22617 raft_consensus.cc:374] T 00000000000000000000000000000000 P f6b682a6ffac4d2789a278e5a2bb5eb1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:48:37.730928 22617 raft_consensus.cc:729] T 00000000000000000000000000000000 P f6b682a6ffac4d2789a278e5a2bb5eb1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6b682a6ffac4d2789a278e5a2bb5eb1, State: Initialized, Role: FOLLOWER
I20260619 16:48:37.731545 22617 consensus_queue.cc:260] T 00000000000000000000000000000000 P f6b682a6ffac4d2789a278e5a2bb5eb1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6b682a6ffac4d2789a278e5a2bb5eb1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37549 } }
I20260619 16:48:37.731797 22617 raft_consensus.cc:388] T 00000000000000000000000000000000 P f6b682a6ffac4d2789a278e5a2bb5eb1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:48:37.731918 22617 raft_consensus.cc:482] T 00000000000000000000000000000000 P f6b682a6ffac4d2789a278e5a2bb5eb1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:48:37.732066 22617 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f6b682a6ffac4d2789a278e5a2bb5eb1 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:48:37.734074 22617 raft_consensus.cc:504] T 00000000000000000000000000000000 P f6b682a6ffac4d2789a278e5a2bb5eb1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6b682a6ffac4d2789a278e5a2bb5eb1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37549 } }
I20260619 16:48:37.734515 22617 leader_election.cc:302] T 00000000000000000000000000000000 P f6b682a6ffac4d2789a278e5a2bb5eb1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f6b682a6ffac4d2789a278e5a2bb5eb1; no voters:
I20260619 16:48:37.735002 22617 leader_election.cc:288] T 00000000000000000000000000000000 P f6b682a6ffac4d2789a278e5a2bb5eb1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 16:48:37.735127 22627 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f6b682a6ffac4d2789a278e5a2bb5eb1 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:48:37.735721 22627 raft_consensus.cc:686] T 00000000000000000000000000000000 P f6b682a6ffac4d2789a278e5a2bb5eb1 [term 1 LEADER]: Becoming Leader. State: Replica: f6b682a6ffac4d2789a278e5a2bb5eb1, State: Running, Role: LEADER
I20260619 16:48:37.736181 22627 consensus_queue.cc:237] T 00000000000000000000000000000000 P f6b682a6ffac4d2789a278e5a2bb5eb1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6b682a6ffac4d2789a278e5a2bb5eb1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37549 } }
I20260619 16:48:37.736735 22617 sys_catalog.cc:564] T 00000000000000000000000000000000 P f6b682a6ffac4d2789a278e5a2bb5eb1 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:48:37.739357 22629 sys_catalog.cc:455] T 00000000000000000000000000000000 P f6b682a6ffac4d2789a278e5a2bb5eb1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f6b682a6ffac4d2789a278e5a2bb5eb1. Latest consensus state: current_term: 1 leader_uuid: "f6b682a6ffac4d2789a278e5a2bb5eb1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6b682a6ffac4d2789a278e5a2bb5eb1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37549 } } }
I20260619 16:48:37.739749 22629 sys_catalog.cc:458] T 00000000000000000000000000000000 P f6b682a6ffac4d2789a278e5a2bb5eb1 [sys.catalog]: This master's current role is: LEADER
I20260619 16:48:37.740054 22628 sys_catalog.cc:455] T 00000000000000000000000000000000 P f6b682a6ffac4d2789a278e5a2bb5eb1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f6b682a6ffac4d2789a278e5a2bb5eb1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6b682a6ffac4d2789a278e5a2bb5eb1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37549 } } }
I20260619 16:48:37.740331 22628 sys_catalog.cc:458] T 00000000000000000000000000000000 P f6b682a6ffac4d2789a278e5a2bb5eb1 [sys.catalog]: This master's current role is: LEADER
I20260619 16:48:37.741611 22636 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:48:37.746282 22636 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:48:37.754063 22636 catalog_manager.cc:1347] Generated new cluster ID: 0b162bae51a440e68afbad02d86454a3
I20260619 16:48:37.754220 22636 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:48:37.764626 22636 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 16:48:37.766379 22636 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:48:37.774691 22636 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f6b682a6ffac4d2789a278e5a2bb5eb1: Generated new TSK 0
I20260619 16:48:37.775630 22636 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 16:48:37.785673 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu pid 22550
W20260619 16:48:37.786661 22612 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:34927
W20260619 16:48:37.787302 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu pid 22550: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 16:48:37.787438 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu with pid 22550
2026-06-19T16:48:37Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1739 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781887385903558-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 16:48:37 dist-test-slave-wvz7 krb5kdc[22648](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 16:48:37 dist-test-slave-wvz7 krb5kdc[22648](info): set up 2 sockets
Jun 19 16:48:37 dist-test-slave-wvz7 krb5kdc[22648](info): commencing operation
krb5kdc: starting...
W20260619 16:48:38.789862 388 mini_kdc.cc:121] Time spent starting KDC: real 0.971s user 0.004s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781887385903558-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781887385903558-388-0/krb5kdc/test-user.keytab.
Jun 19 16:48:38 dist-test-slave-wvz7 krb5kdc[22648](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887718, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T16:48:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T16:48:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781887385903558-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781887385903558-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260619 16:48:39.012377 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu
/tmp/dist-test-taskEs2EH8/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781887385903558-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781887385903558-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781887385903558-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781887385903558-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37675
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781887385903558-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45619
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:37675
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781887385903558-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781887385903558-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781887385903558-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:48:39.323495 22661 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:48:39.323770 22661 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:48:39.323905 22661 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:48:39.330835 22661 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:48:39.330942 22661 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:48:39.331014 22661 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:48:39.331092 22661 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:48:39.331151 22661 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:48:39.340794 22661 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45619
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781887385903558-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781887385903558-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:37675
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781887385903558-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37675
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781887385903558-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781887385903558-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:48:39.342056 22661 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:48:39.343315 22661 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:48:39.354477 22669 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:48:39.354725 22666 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:48:39.355513 22661 server_base.cc:1034] running on GCE node
W20260619 16:48:39.356060 22667 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:48:39.357388 22661 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:48:39.360582 22661 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:48:39.361824 22661 hybrid_clock.cc:648] HybridClock initialized: now 1781887719361767 us; error 45 us; skew 500 ppm
I20260619 16:48:39.362182 22661 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:48:39 dist-test-slave-wvz7 krb5kdc[22648](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887719, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:48:39.365134 22661 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260619 16:48:39.366521 22661 webserver.cc:460] Webserver started at http://127.0.97.62:34893/ using document root <none> and password file <none>
I20260619 16:48:39.367161 22661 fs_manager.cc:359] Metadata directory not provided
I20260619 16:48:39.367271 22661 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:48:39.367547 22661 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:48:39.370385 22661 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781887385903558-388-0/minicluster-data/master-0/data/instance:
uuid: "767b7def55eb4967a8f50c842217887f"
format_stamp: "Formatted at 2026-06-19 16:48:39 on dist-test-slave-wvz7"
server_key: "18866d76be23e995dbfe0dc7fe663ba9"
server_key_iv: "823dd699ca7ae0534985028e6b1c535d"
server_key_version: "encryptionkey@0"
I20260619 16:48:39.371073 22661 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781887385903558-388-0/minicluster-data/master-0/wal/instance:
uuid: "767b7def55eb4967a8f50c842217887f"
format_stamp: "Formatted at 2026-06-19 16:48:39 on dist-test-slave-wvz7"
server_key: "18866d76be23e995dbfe0dc7fe663ba9"
server_key_iv: "823dd699ca7ae0534985028e6b1c535d"
server_key_version: "encryptionkey@0"
I20260619 16:48:39.375972 22661 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260619 16:48:39.379971 22676 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:48:39.381228 22661 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 16:48:39.381378 22661 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781887385903558-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781887385903558-388-0/minicluster-data/master-0/wal
uuid: "767b7def55eb4967a8f50c842217887f"
format_stamp: "Formatted at 2026-06-19 16:48:39 on dist-test-slave-wvz7"
server_key: "18866d76be23e995dbfe0dc7fe663ba9"
server_key_iv: "823dd699ca7ae0534985028e6b1c535d"
server_key_version: "encryptionkey@0"
I20260619 16:48:39.381551 22661 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781887385903558-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781887385903558-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781887385903558-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:48:39.386893 22661 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:48:39.390782 22661 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:48:39.391026 22661 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:48:39.407032 22661 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:37675
I20260619 16:48:39.407066 22727 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:37675 every 8 connection(s)
I20260619 16:48:39.408278 22661 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781887385903558-388-0/minicluster-data/master-0/data/info.pb
I20260619 16:48:39.411675 22728 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:48:39.414505 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu as pid 22661
I20260619 16:48:39.414674 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781887385903558-388-0/minicluster-data/master-0/wal/instance
I20260619 16:48:39.415182 388 external_mini_cluster.cc:1428] Setting key 32ac475c9409c3bff1d427edd44c1183
I20260619 16:48:39.422780 22731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:48:39.416955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33103 (local address 127.0.97.62:37675)
0619 16:48:39.417762 (+ 807us) server_negotiation.cc:206] Beginning negotiation
0619 16:48:39.417777 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 16:48:39.417819 (+ 42us) server_negotiation.cc:407] Connection header received
0619 16:48:39.419545 (+ 1726us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:48:39.419567 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:48:39.419929 (+ 362us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:48:39.420261 (+ 332us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:48:39.421269 (+ 1008us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:48:39.421276 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:48:39.421289 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:48:39.421304 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:48:39.421369 (+ 65us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:48:39.421391 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:48:39.421566 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0619 16:48:39.421675 (+ 109us) server_negotiation.cc:299] Negotiation successful
0619 16:48:39.421893 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":551,"thread_start_us":346,"threads_started":1}
I20260619 16:48:39.423087 22728 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 767b7def55eb4967a8f50c842217887f: Bootstrap starting.
I20260619 16:48:39.426368 22728 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 767b7def55eb4967a8f50c842217887f: Neither blocks nor log segments found. Creating new log.
I20260619 16:48:39.426429 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 16:48:39.427491 22728 log.cc:824] T 00000000000000000000000000000000 P 767b7def55eb4967a8f50c842217887f: Log is configured to *not* fsync() on all Append() calls
I20260619 16:48:39.430521 22728 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 767b7def55eb4967a8f50c842217887f: No bootstrap required, opened a new log
I20260619 16:48:39.433668 22731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:48:39.431357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54641 (local address 127.0.97.62:37675)
0619 16:48:39.431528 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0619 16:48:39.431533 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:48:39.431853 (+ 320us) server_negotiation.cc:407] Connection header received
0619 16:48:39.432007 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:48:39.432010 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:48:39.432056 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:48:39.432140 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:48:39.433041 (+ 901us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:48:39.433046 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:48:39.433049 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:48:39.433064 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:48:39.433103 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:48:39.433117 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:48:39.433215 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0619 16:48:39.433359 (+ 144us) server_negotiation.cc:299] Negotiation successful
0619 16:48:39.433407 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260619 16:48:39.438529 22728 raft_consensus.cc:348] T 00000000000000000000000000000000 P 767b7def55eb4967a8f50c842217887f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "767b7def55eb4967a8f50c842217887f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37675 } }
I20260619 16:48:39.438921 22728 raft_consensus.cc:374] T 00000000000000000000000000000000 P 767b7def55eb4967a8f50c842217887f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:48:39.439038 22728 raft_consensus.cc:729] T 00000000000000000000000000000000 P 767b7def55eb4967a8f50c842217887f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 767b7def55eb4967a8f50c842217887f, State: Initialized, Role: FOLLOWER
I20260619 16:48:39.439563 22728 consensus_queue.cc:260] T 00000000000000000000000000000000 P 767b7def55eb4967a8f50c842217887f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "767b7def55eb4967a8f50c842217887f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37675 } }
I20260619 16:48:39.439769 22728 raft_consensus.cc:388] T 00000000000000000000000000000000 P 767b7def55eb4967a8f50c842217887f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:48:39.439893 22728 raft_consensus.cc:482] T 00000000000000000000000000000000 P 767b7def55eb4967a8f50c842217887f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:48:39.440035 22728 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 767b7def55eb4967a8f50c842217887f [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:48:39.442086 22728 raft_consensus.cc:504] T 00000000000000000000000000000000 P 767b7def55eb4967a8f50c842217887f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "767b7def55eb4967a8f50c842217887f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37675 } }
I20260619 16:48:39.442528 22728 leader_election.cc:302] T 00000000000000000000000000000000 P 767b7def55eb4967a8f50c842217887f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 767b7def55eb4967a8f50c842217887f; no voters:
I20260619 16:48:39.443100 22728 leader_election.cc:288] T 00000000000000000000000000000000 P 767b7def55eb4967a8f50c842217887f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 16:48:39.443202 22738 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 767b7def55eb4967a8f50c842217887f [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:48:39.443764 22738 raft_consensus.cc:686] T 00000000000000000000000000000000 P 767b7def55eb4967a8f50c842217887f [term 1 LEADER]: Becoming Leader. State: Replica: 767b7def55eb4967a8f50c842217887f, State: Running, Role: LEADER
I20260619 16:48:39.444335 22738 consensus_queue.cc:237] T 00000000000000000000000000000000 P 767b7def55eb4967a8f50c842217887f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "767b7def55eb4967a8f50c842217887f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37675 } }
I20260619 16:48:39.445128 22728 sys_catalog.cc:564] T 00000000000000000000000000000000 P 767b7def55eb4967a8f50c842217887f [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:48:39.447119 22739 sys_catalog.cc:455] T 00000000000000000000000000000000 P 767b7def55eb4967a8f50c842217887f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "767b7def55eb4967a8f50c842217887f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "767b7def55eb4967a8f50c842217887f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37675 } } }
I20260619 16:48:39.447510 22739 sys_catalog.cc:458] T 00000000000000000000000000000000 P 767b7def55eb4967a8f50c842217887f [sys.catalog]: This master's current role is: LEADER
I20260619 16:48:39.448170 22740 sys_catalog.cc:455] T 00000000000000000000000000000000 P 767b7def55eb4967a8f50c842217887f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 767b7def55eb4967a8f50c842217887f. Latest consensus state: current_term: 1 leader_uuid: "767b7def55eb4967a8f50c842217887f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "767b7def55eb4967a8f50c842217887f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37675 } } }
I20260619 16:48:39.448590 22740 sys_catalog.cc:458] T 00000000000000000000000000000000 P 767b7def55eb4967a8f50c842217887f [sys.catalog]: This master's current role is: LEADER
I20260619 16:48:39.448979 22743 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:48:39.453992 22743 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:48:39.463063 22743 catalog_manager.cc:1347] Generated new cluster ID: 2f3c15e849af443ba53e821761e2a685
I20260619 16:48:39.463167 22743 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:48:39.481848 22743 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 16:48:39.482205 22743 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:48:39.492666 22743 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 767b7def55eb4967a8f50c842217887f: Generated new TSK 0
I20260619 16:48:39.493445 22743 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 16:48:39.509613 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu pid 22661
W20260619 16:48:39.510650 22723 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:33103
W20260619 16:48:39.511291 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu pid 22661: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 16:48:39.511418 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu with pid 22661
2026-06-19T16:48:39Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1723 ms)
[----------] 18 tests from AuthTokenIssuingTest (32318 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887385903558-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 16:48:39 dist-test-slave-wvz7 krb5kdc[22759](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 16:48:39 dist-test-slave-wvz7 krb5kdc[22759](info): set up 2 sockets
Jun 19 16:48:39 dist-test-slave-wvz7 krb5kdc[22759](info): commencing operation
krb5kdc: starting...
W20260619 16:48:40.514076 388 mini_kdc.cc:121] Time spent starting KDC: real 0.971s user 0.004s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887385903558-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887385903558-388-0/krb5kdc/test-user.keytab.
Jun 19 16:48:40 dist-test-slave-wvz7 krb5kdc[22759](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887720, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T16:48:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T16:48:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887385903558-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887385903558-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260619 16:48:40.734480 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu
/tmp/dist-test-taskEs2EH8/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887385903558-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887385903558-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887385903558-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887385903558-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35423
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887385903558-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45017
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:35423,127.0.97.61:40923,127.0.97.60:38283
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887385903558-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887385903558-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887385903558-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:48:41.038401 22772 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260619 16:48:41.038748 22772 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:48:41.038834 22772 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:48:41.038915 22772 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:48:41.046255 22772 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:48:41.046352 22772 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:48:41.046408 22772 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:48:41.046458 22772 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:48:41.046516 22772 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:48:41.056330 22772 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45017
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887385903558-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887385903558-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:35423,127.0.97.61:40923,127.0.97.60:38283
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887385903558-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35423
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887385903558-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887385903558-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:48:41.057523 22772 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:48:41.058732 22772 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:48:41.069746 22778 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:48:41.069727 22777 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:48:41.069922 22780 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:48:41.070556 22772 server_base.cc:1034] running on GCE node
I20260619 16:48:41.071290 22772 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:48:41.072638 22772 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:48:41.073858 22772 hybrid_clock.cc:648] HybridClock initialized: now 1781887721073778 us; error 49 us; skew 500 ppm
I20260619 16:48:41.074158 22772 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:48:41 dist-test-slave-wvz7 krb5kdc[22759](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887721, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:48:41.077080 22772 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260619 16:48:41.078447 22772 webserver.cc:460] Webserver started at http://127.0.97.62:35805/ using document root <none> and password file <none>
I20260619 16:48:41.079101 22772 fs_manager.cc:359] Metadata directory not provided
I20260619 16:48:41.079200 22772 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:48:41.079497 22772 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:48:41.082381 22772 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887385903558-388-0/minicluster-data/master-0/data/instance:
uuid: "5b699388e5ad4dcda958440823bf99a8"
format_stamp: "Formatted at 2026-06-19 16:48:41 on dist-test-slave-wvz7"
server_key: "193e7504fa972b6e739d1576ab1cbb27"
server_key_iv: "7da6f467b828be910490464b06e9b72f"
server_key_version: "encryptionkey@0"
I20260619 16:48:41.083141 22772 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887385903558-388-0/minicluster-data/master-0/wal/instance:
uuid: "5b699388e5ad4dcda958440823bf99a8"
format_stamp: "Formatted at 2026-06-19 16:48:41 on dist-test-slave-wvz7"
server_key: "193e7504fa972b6e739d1576ab1cbb27"
server_key_iv: "7da6f467b828be910490464b06e9b72f"
server_key_version: "encryptionkey@0"
I20260619 16:48:41.087929 22772 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 16:48:41.091254 22787 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:48:41.092576 22772 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 16:48:41.092770 22772 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887385903558-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887385903558-388-0/minicluster-data/master-0/wal
uuid: "5b699388e5ad4dcda958440823bf99a8"
format_stamp: "Formatted at 2026-06-19 16:48:41 on dist-test-slave-wvz7"
server_key: "193e7504fa972b6e739d1576ab1cbb27"
server_key_iv: "7da6f467b828be910490464b06e9b72f"
server_key_version: "encryptionkey@0"
I20260619 16:48:41.092976 22772 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887385903558-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887385903558-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887385903558-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:48:41.121145 22772 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:48:41.125257 22772 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:48:41.125506 22772 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:48:41.142380 22772 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:35423
I20260619 16:48:41.142400 22838 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:35423 every 8 connection(s)
I20260619 16:48:41.143637 22772 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887385903558-388-0/minicluster-data/master-0/data/info.pb
I20260619 16:48:41.144361 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu as pid 22772
I20260619 16:48:41.144575 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887385903558-388-0/minicluster-data/master-0/wal/instance
I20260619 16:48:41.145301 388 external_mini_cluster.cc:1428] Setting key 33145f2ed0bd014459b73f5c8136910d
I20260619 16:48:41.147501 22839 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:48:41.153951 22839 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35423 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.97.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.61@KRBTEST.COM" created.
Jun 19 16:48:41 dist-test-slave-wvz7 krb5kdc[22759](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887721, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 16:48:41.176484 22842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:48:41.156833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35423 (local address 127.0.0.1:46260)
0619 16:48:41.157611 (+ 778us) negotiation.cc:107] Waiting for socket to connect
0619 16:48:41.157671 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0619 16:48:41.159665 (+ 1994us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:48:41.160579 (+ 914us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:48:41.160584 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:48:41.161155 (+ 571us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:48:41.161500 (+ 345us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:48:41.161547 (+ 47us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:48:41.163806 (+ 2259us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:48:41.163812 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:48:41.165173 (+ 1361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:48:41.165184 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:48:41.166451 (+ 1267us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:48:41.166457 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:48:41.166597 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:48:41.167181 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:48:41.167210 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:48:41.169530 (+ 2320us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:48:41.172525 (+ 2995us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:48:41.172535 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:48:41.172547 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:48:41.172896 (+ 349us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:48:41.173260 (+ 364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:48:41.173264 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:48:41.173266 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:48:41.173412 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:48:41.174182 (+ 770us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:48:41.174190 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:48:41.174377 (+ 187us) client_negotiation.cc:769] Sending connection context
0619 16:48:41.174594 (+ 217us) client_negotiation.cc:240] Negotiation successful
0619 16:48:41.175201 (+ 607us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":515,"mutex_wait_us":128,"thread_start_us":323,"threads_started":1}
I20260619 16:48:41.176517 22843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:48:41.158095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46260 (local address 127.0.97.62:35423)
0619 16:48:41.158737 (+ 642us) server_negotiation.cc:206] Beginning negotiation
0619 16:48:41.158745 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 16:48:41.158794 (+ 49us) server_negotiation.cc:407] Connection header received
0619 16:48:41.160033 (+ 1239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:48:41.160040 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:48:41.160348 (+ 308us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:48:41.160548 (+ 200us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:48:41.161772 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:48:41.163601 (+ 1829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:48:41.165311 (+ 1710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:48:41.166220 (+ 909us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:48:41.166419 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:48:41.169759 (+ 3340us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:48:41.169771 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:48:41.169787 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:48:41.169816 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:48:41.172400 (+ 2584us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:48:41.173012 (+ 612us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:48:41.173018 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:48:41.173020 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:48:41.173153 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:48:41.173507 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:48:41.173511 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:48:41.173513 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:48:41.174042 (+ 529us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:48:41.174214 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0619 16:48:41.174742 (+ 528us) server_negotiation.cc:299] Negotiation successful
0619 16:48:41.175068 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":47,"server-negotiator.queue_time_us":457,"thread_start_us":312,"threads_started":1}
I20260619 16:48:41.179865 22839 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40923 } has no permanent_uuid. Determining permanent_uuid...
I20260619 16:48:41.180998 22842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:48:41.180557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:40923 (local address 127.0.0.1:44038)
0619 16:48:41.180733 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0619 16:48:41.180856 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:40923: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260619 16:48:41.181301 22791 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.61:40923: connect: Connection refused (error 111)
I20260619 16:48:41.182868 22842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:48:41.182478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:40923 (local address 127.0.0.1:44044)
0619 16:48:41.182622 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0619 16:48:41.182710 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:40923: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260619 16:48:41.183068 22839 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:40923: Network error: Client connection negotiation failed: client connection to 127.0.97.61:40923: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887385903558-388-0/krb5kdc/kudu_127.0.97.61.keytab.
Entry for principal kudu/127.0.97.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887385903558-388-0/krb5kdc/kudu_127.0.97.61.keytab.
I20260619 16:48:41.195914 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu
/tmp/dist-test-taskEs2EH8/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887385903558-388-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887385903558-388-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887385903558-388-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887385903558-388-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:40923
--webserver_interface=127.0.97.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887385903558-388-0/krb5kdc/kudu_127.0.97.61.keytab
--principal=kudu/127.0.97.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45017
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:35423,127.0.97.61:40923,127.0.97.60:38283
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887385903558-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887385903558-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887385903558-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 16:48:41.232414 22839 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40923 } attempt: 1
I20260619 16:48:41.233855 22842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:48:41.233243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:40923 (local address 127.0.0.1:44060)
0619 16:48:41.233473 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0619 16:48:41.233608 (+ 135us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:40923: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260619 16:48:41.234958 22842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:48:41.234525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:40923 (local address 127.0.0.1:44070)
0619 16:48:41.234688 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0619 16:48:41.234788 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:40923: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260619 16:48:41.235186 22839 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:40923: Network error: Client connection negotiation failed: client connection to 127.0.97.61:40923: connect: Connection refused (error 111)
I20260619 16:48:41.303447 22839 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40923 } attempt: 2
I20260619 16:48:41.304673 22842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:48:41.304121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:40923 (local address 127.0.0.1:44084)
0619 16:48:41.304380 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0619 16:48:41.304495 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:40923: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260619 16:48:41.305701 22842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:48:41.305294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:40923 (local address 127.0.0.1:44090)
0619 16:48:41.305446 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0619 16:48:41.305547 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:40923: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260619 16:48:41.305948 22839 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:40923: Network error: Client connection negotiation failed: client connection to 127.0.97.61:40923: connect: Connection refused (error 111)
I20260619 16:48:41.397203 22839 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40923 } attempt: 3
I20260619 16:48:41.398393 22842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:48:41.397898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:40923 (local address 127.0.0.1:44106)
0619 16:48:41.398110 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0619 16:48:41.398222 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:40923: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260619 16:48:41.399420 22842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:48:41.399011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:40923 (local address 127.0.0.1:44114)
0619 16:48:41.399168 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0619 16:48:41.399255 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:40923: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260619 16:48:41.399608 22839 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:40923: Network error: Client connection negotiation failed: client connection to 127.0.97.61:40923: connect: Connection refused (error 111)
W20260619 16:48:41.508503 22846 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260619 16:48:41.508805 22846 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:48:41.508901 22846 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:48:41.508991 22846 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:48:41.516090 22846 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:48:41.516211 22846 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:48:41.516306 22846 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:48:41.516369 22846 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:48:41.516451 22846 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:48:41.525981 22846 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45017
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887385903558-388-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887385903558-388-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:35423,127.0.97.61:40923,127.0.97.60:38283
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887385903558-388-0/krb5kdc/kudu_127.0.97.61.keytab
--principal=kudu/127.0.97.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:40923
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887385903558-388-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.97.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887385903558-388-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:48:41.527189 22846 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:48:41.528463 22846 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:48:41.539069 22851 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:48:41.539170 22854 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:48:41.539235 22852 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:48:41.539842 22846 server_base.cc:1034] running on GCE node
I20260619 16:48:41.540577 22846 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:48:41.541940 22846 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:48:41.542809 22839 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40923 } attempt: 4
I20260619 16:48:41.543174 22846 hybrid_clock.cc:648] HybridClock initialized: now 1781887721543101 us; error 40 us; skew 500 ppm
I20260619 16:48:41.543478 22846 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 16:48:41.543980 22842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:48:41.543393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:40923 (local address 127.0.0.1:44126)
0619 16:48:41.543719 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0619 16:48:41.543822 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:40923: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135}
I20260619 16:48:41.544931 22842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:48:41.544592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:40923 (local address 127.0.0.1:44136)
0619 16:48:41.544738 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0619 16:48:41.544817 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:40923: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260619 16:48:41.545113 22839 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:40923: Network error: Client connection negotiation failed: client connection to 127.0.97.61:40923: connect: Connection refused (error 111)
Jun 19 16:48:41 dist-test-slave-wvz7 krb5kdc[22759](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887721, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:48:41.547005 22846 init.cc:377] Logged in from keytab as kudu/127.0.97.61@KRBTEST.COM (short username kudu)
I20260619 16:48:41.548398 22846 webserver.cc:460] Webserver started at http://127.0.97.61:41015/ using document root <none> and password file <none>
I20260619 16:48:41.549057 22846 fs_manager.cc:359] Metadata directory not provided
I20260619 16:48:41.549144 22846 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:48:41.549443 22846 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:48:41.552395 22846 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887385903558-388-0/minicluster-data/master-1/data/instance:
uuid: "f2c48d2a6aa248ed880e12de4b6b18e5"
format_stamp: "Formatted at 2026-06-19 16:48:41 on dist-test-slave-wvz7"
server_key: "b51ae05d1bcf8c731a2fb5e66b4be342"
server_key_iv: "4f1e1225c5e977ee39f8937a6d817832"
server_key_version: "encryptionkey@0"
I20260619 16:48:41.553134 22846 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887385903558-388-0/minicluster-data/master-1/wal/instance:
uuid: "f2c48d2a6aa248ed880e12de4b6b18e5"
format_stamp: "Formatted at 2026-06-19 16:48:41 on dist-test-slave-wvz7"
server_key: "b51ae05d1bcf8c731a2fb5e66b4be342"
server_key_iv: "4f1e1225c5e977ee39f8937a6d817832"
server_key_version: "encryptionkey@0"
I20260619 16:48:41.558058 22846 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260619 16:48:41.561291 22861 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:48:41.562583 22846 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 16:48:41.562744 22846 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887385903558-388-0/minicluster-data/master-1/data,/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887385903558-388-0/minicluster-data/master-1/wal
uuid: "f2c48d2a6aa248ed880e12de4b6b18e5"
format_stamp: "Formatted at 2026-06-19 16:48:41 on dist-test-slave-wvz7"
server_key: "b51ae05d1bcf8c731a2fb5e66b4be342"
server_key_iv: "4f1e1225c5e977ee39f8937a6d817832"
server_key_version: "encryptionkey@0"
I20260619 16:48:41.562932 22846 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887385903558-388-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887385903558-388-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887385903558-388-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:48:41.580426 22846 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:48:41.584026 22846 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:48:41.584249 22846 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:48:41.600289 22846 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.61:40923
I20260619 16:48:41.600311 22912 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.61:40923 every 8 connection(s)
I20260619 16:48:41.601508 22846 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887385903558-388-0/minicluster-data/master-1/data/info.pb
I20260619 16:48:41.604982 22913 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:48:41.607843 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu as pid 22846
I20260619 16:48:41.608026 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887385903558-388-0/minicluster-data/master-1/wal/instance
I20260619 16:48:41.608456 388 external_mini_cluster.cc:1428] Setting key 9f30ca7731e5a65930059fcc4161c968
I20260619 16:48:41.611150 22913 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35423 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 16:48:41 dist-test-slave-wvz7 krb5kdc[22759](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887721, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.60@KRBTEST.COM; defaulting to no policy
I20260619 16:48:41.631434 22843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:48:41.613544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46274 (local address 127.0.97.62:35423)
0619 16:48:41.613793 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0619 16:48:41.613800 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:48:41.614982 (+ 1182us) server_negotiation.cc:407] Connection header received
0619 16:48:41.617083 (+ 2101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:48:41.617089 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:48:41.617168 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:48:41.617307 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:48:41.618806 (+ 1499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:48:41.619688 (+ 882us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:48:41.621194 (+ 1506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:48:41.621761 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:48:41.621875 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:48:41.625349 (+ 3474us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:48:41.625353 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:48:41.625357 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:48:41.625390 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:48:41.628248 (+ 2858us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:48:41.629109 (+ 861us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:48:41.629114 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:48:41.629116 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:48:41.629270 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:48:41.629727 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:48:41.629755 (+ 28us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:48:41.629758 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:48:41.630056 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:48:41.630207 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0619 16:48:41.630883 (+ 676us) server_negotiation.cc:299] Negotiation successful
0619 16:48:41.631109 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.60@KRBTEST.COM" created.
I20260619 16:48:41.632586 22915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:48:41.613998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35423 (local address 127.0.0.1:46274)
0619 16:48:41.614802 (+ 804us) negotiation.cc:107] Waiting for socket to connect
0619 16:48:41.614853 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0619 16:48:41.616808 (+ 1955us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:48:41.617484 (+ 676us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:48:41.617492 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:48:41.618064 (+ 572us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:48:41.618645 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:48:41.618663 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:48:41.619855 (+ 1192us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:48:41.619859 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:48:41.621060 (+ 1201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:48:41.621071 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:48:41.621938 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:48:41.621943 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:48:41.622131 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:48:41.622809 (+ 678us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:48:41.622831 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:48:41.625112 (+ 2281us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:48:41.628395 (+ 3283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:48:41.628403 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:48:41.628421 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:48:41.628982 (+ 561us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:48:41.629380 (+ 398us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:48:41.629386 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:48:41.629389 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:48:41.629612 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:48:41.630169 (+ 557us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:48:41.630174 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:48:41.630490 (+ 316us) client_negotiation.cc:769] Sending connection context
0619 16:48:41.630772 (+ 282us) client_negotiation.cc:240] Negotiation successful
0619 16:48:41.631269 (+ 497us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":511,"thread_start_us":302,"threads_started":1}
I20260619 16:48:41.634371 22913 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40923 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 16:48:41 dist-test-slave-wvz7 krb5kdc[22759](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887721, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260619 16:48:41.648065 22915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:48:41.635174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:40923 (local address 127.0.0.1:44140)
0619 16:48:41.635376 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0619 16:48:41.635392 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 16:48:41.635504 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:48:41.636171 (+ 667us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:48:41.636175 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:48:41.636575 (+ 400us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:48:41.636693 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:48:41.636701 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:48:41.638049 (+ 1348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:48:41.638053 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:48:41.638902 (+ 849us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:48:41.638913 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:48:41.639657 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:48:41.639663 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:48:41.639833 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:48:41.640338 (+ 505us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:48:41.640354 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:48:41.642524 (+ 2170us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:48:41.645515 (+ 2991us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:48:41.645520 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:48:41.645523 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:48:41.646043 (+ 520us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:48:41.646409 (+ 366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:48:41.646413 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:48:41.646417 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:48:41.646551 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:48:41.647124 (+ 573us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:48:41.647129 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:48:41.647283 (+ 154us) client_negotiation.cc:769] Sending connection context
0619 16:48:41.647415 (+ 132us) client_negotiation.cc:240] Negotiation successful
0619 16:48:41.647710 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":77}
I20260619 16:48:41.648094 22917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:48:41.635243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44140 (local address 127.0.97.61:40923)
0619 16:48:41.635798 (+ 555us) server_negotiation.cc:206] Beginning negotiation
0619 16:48:41.635804 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:48:41.635825 (+ 21us) server_negotiation.cc:407] Connection header received
0619 16:48:41.635864 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:48:41.635870 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:48:41.636040 (+ 170us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:48:41.636170 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:48:41.636845 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:48:41.637908 (+ 1063us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:48:41.639018 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:48:41.639531 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:48:41.639637 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:48:41.642733 (+ 3096us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:48:41.642753 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:48:41.642766 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:48:41.642794 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:48:41.645394 (+ 2600us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:48:41.646155 (+ 761us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:48:41.646162 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:48:41.646165 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:48:41.646300 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:48:41.646647 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:48:41.646652 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:48:41.646656 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:48:41.647003 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:48:41.647159 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0619 16:48:41.647566 (+ 407us) server_negotiation.cc:299] Negotiation successful
0619 16:48:41.647814 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":56,"server-negotiator.queue_time_us":416,"thread_start_us":277,"threads_started":1}
I20260619 16:48:41.650761 22913 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 38283 } has no permanent_uuid. Determining permanent_uuid...
I20260619 16:48:41.651844 22915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:48:41.651365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:38283 (local address 127.0.0.1:58026)
0619 16:48:41.651526 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0619 16:48:41.651656 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:38283: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260619 16:48:41.652088 22865 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.60:38283: connect: Connection refused (error 111)
I20260619 16:48:41.653663 22915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:48:41.653319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:38283 (local address 127.0.0.1:58034)
0619 16:48:41.653477 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0619 16:48:41.653569 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:38283: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260619 16:48:41.653898 22913 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:38283: Network error: Client connection negotiation failed: client connection to 127.0.97.60:38283: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887385903558-388-0/krb5kdc/kudu_127.0.97.60.keytab.
Entry for principal kudu/127.0.97.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887385903558-388-0/krb5kdc/kudu_127.0.97.60.keytab.
I20260619 16:48:41.660064 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu
/tmp/dist-test-taskEs2EH8/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887385903558-388-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887385903558-388-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887385903558-388-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887385903558-388-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:38283
--webserver_interface=127.0.97.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887385903558-388-0/krb5kdc/kudu_127.0.97.60.keytab
--principal=kudu/127.0.97.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45017
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:35423,127.0.97.61:40923,127.0.97.60:38283
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887385903558-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887385903558-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887385903558-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 16:48:41.703188 22913 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 38283 } attempt: 1
I20260619 16:48:41.704312 22915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:48:41.703778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:38283 (local address 127.0.0.1:58046)
0619 16:48:41.703996 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0619 16:48:41.704110 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:38283: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260619 16:48:41.705376 22915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:48:41.704952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:38283 (local address 127.0.0.1:58060)
0619 16:48:41.705117 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0619 16:48:41.705239 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:38283: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260619 16:48:41.705588 22913 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:38283: Network error: Client connection negotiation failed: client connection to 127.0.97.60:38283: connect: Connection refused (error 111)
I20260619 16:48:41.773830 22913 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 38283 } attempt: 2
I20260619 16:48:41.775029 22915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:48:41.774509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:38283 (local address 127.0.0.1:58074)
0619 16:48:41.774722 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0619 16:48:41.774846 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:38283: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260619 16:48:41.775992 22915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:48:41.775622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:38283 (local address 127.0.0.1:58088)
0619 16:48:41.775761 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0619 16:48:41.775839 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:38283: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260619 16:48:41.776191 22913 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:38283: Network error: Client connection negotiation failed: client connection to 127.0.97.60:38283: connect: Connection refused (error 111)
I20260619 16:48:41.844348 22839 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40923 } attempt: 5
Jun 19 16:48:41 dist-test-slave-wvz7 krb5kdc[22759](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887721, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260619 16:48:41.856981 22842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:48:41.845052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:40923 (local address 127.0.0.1:44142)
0619 16:48:41.845259 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0619 16:48:41.845281 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0619 16:48:41.845411 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:48:41.845782 (+ 371us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:48:41.845787 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:48:41.846177 (+ 390us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:48:41.846353 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:48:41.846362 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:48:41.847549 (+ 1187us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:48:41.847553 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:48:41.848375 (+ 822us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:48:41.848386 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:48:41.849074 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:48:41.849079 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:48:41.849260 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:48:41.849846 (+ 586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:48:41.849862 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:48:41.852074 (+ 2212us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:48:41.854790 (+ 2716us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:48:41.854795 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:48:41.854800 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:48:41.855155 (+ 355us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:48:41.855485 (+ 330us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:48:41.855490 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:48:41.855494 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:48:41.855625 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:48:41.856092 (+ 467us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:48:41.856097 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:48:41.856247 (+ 150us) client_negotiation.cc:769] Sending connection context
0619 16:48:41.856366 (+ 119us) client_negotiation.cc:240] Negotiation successful
0619 16:48:41.856620 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":67}
I20260619 16:48:41.856981 22917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:48:41.845161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44142 (local address 127.0.97.61:40923)
0619 16:48:41.845333 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0619 16:48:41.845337 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:48:41.845359 (+ 22us) server_negotiation.cc:407] Connection header received
0619 16:48:41.845539 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:48:41.845544 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:48:41.845624 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:48:41.845708 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:48:41.846503 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:48:41.847402 (+ 899us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:48:41.848520 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:48:41.848941 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:48:41.849047 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:48:41.852249 (+ 3202us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:48:41.852253 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:48:41.852256 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:48:41.852288 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:48:41.854674 (+ 2386us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:48:41.855259 (+ 585us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:48:41.855263 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:48:41.855266 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:48:41.855386 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:48:41.855719 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:48:41.855724 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:48:41.855728 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:48:41.855985 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:48:41.856113 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0619 16:48:41.856484 (+ 371us) server_negotiation.cc:299] Negotiation successful
0619 16:48:41.856693 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260619 16:48:41.858104 22839 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 38283 } has no permanent_uuid. Determining permanent_uuid...
I20260619 16:48:41.859016 22842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:48:41.858682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:38283 (local address 127.0.0.1:58090)
0619 16:48:41.858820 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0619 16:48:41.858889 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:38283: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260619 16:48:41.859831 22842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:48:41.859540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:38283 (local address 127.0.0.1:58098)
0619 16:48:41.859663 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0619 16:48:41.859743 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:38283: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260619 16:48:41.860013 22839 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:38283: Network error: Client connection negotiation failed: client connection to 127.0.97.60:38283: connect: Connection refused (error 111)
I20260619 16:48:41.867379 22913 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 38283 } attempt: 3
I20260619 16:48:41.868271 22915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:48:41.867830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:38283 (local address 127.0.0.1:58112)
0619 16:48:41.868015 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0619 16:48:41.868104 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:38283: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260619 16:48:41.869223 22915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:48:41.868796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:38283 (local address 127.0.0.1:58116)
0619 16:48:41.869006 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0619 16:48:41.869088 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:38283: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115}
W20260619 16:48:41.869406 22913 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:38283: Network error: Client connection negotiation failed: client connection to 127.0.97.60:38283: connect: Connection refused (error 111)
I20260619 16:48:41.911209 22839 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 38283 } attempt: 1
I20260619 16:48:41.912256 22842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:48:41.911799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:38283 (local address 127.0.0.1:58120)
0619 16:48:41.911984 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0619 16:48:41.912067 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:38283: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260619 16:48:41.913326 22842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:48:41.912976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:38283 (local address 127.0.0.1:58126)
0619 16:48:41.913134 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0619 16:48:41.913204 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:38283: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260619 16:48:41.913535 22839 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:38283: Network error: Client connection negotiation failed: client connection to 127.0.97.60:38283: connect: Connection refused (error 111)
W20260619 16:48:41.968302 22920 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260619 16:48:41.968603 22920 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:48:41.968678 22920 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:48:41.968760 22920 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:48:41.975754 22920 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:48:41.975839 22920 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:48:41.975894 22920 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:48:41.975940 22920 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:48:41.975998 22920 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:48:41.981769 22839 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 38283 } attempt: 2
I20260619 16:48:41.982880 22842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:48:41.982426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:38283 (local address 127.0.0.1:58138)
0619 16:48:41.982623 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0619 16:48:41.982708 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:38283: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260619 16:48:41.983771 22842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:48:41.983435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:38283 (local address 127.0.0.1:58152)
0619 16:48:41.983594 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0619 16:48:41.983666 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:38283: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260619 16:48:41.983971 22839 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:38283: Network error: Client connection negotiation failed: client connection to 127.0.97.60:38283: connect: Connection refused (error 111)
I20260619 16:48:41.985853 22920 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45017
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887385903558-388-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887385903558-388-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:35423,127.0.97.61:40923,127.0.97.60:38283
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887385903558-388-0/krb5kdc/kudu_127.0.97.60.keytab
--principal=kudu/127.0.97.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:38283
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887385903558-388-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.97.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887385903558-388-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:48:41.987040 22920 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:48:41.988237 22920 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:48:41.998939 22925 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:48:41.999006 22928 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:48:41.998976 22926 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:48:41.999836 22920 server_base.cc:1034] running on GCE node
I20260619 16:48:42.000487 22920 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:48:42.001797 22920 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:48:42.003036 22920 hybrid_clock.cc:648] HybridClock initialized: now 1781887722002973 us; error 34 us; skew 500 ppm
I20260619 16:48:42.003329 22920 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:48:41 dist-test-slave-wvz7 krb5kdc[22759](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887721, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:48:42.006181 22920 init.cc:377] Logged in from keytab as kudu/127.0.97.60@KRBTEST.COM (short username kudu)
I20260619 16:48:42.007618 22920 webserver.cc:460] Webserver started at http://127.0.97.60:39727/ using document root <none> and password file <none>
I20260619 16:48:42.008239 22920 fs_manager.cc:359] Metadata directory not provided
I20260619 16:48:42.008320 22920 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:48:42.008576 22920 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:48:42.011430 22920 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887385903558-388-0/minicluster-data/master-2/data/instance:
uuid: "3bbfc14275c94e54ae7c59dce399387f"
format_stamp: "Formatted at 2026-06-19 16:48:42 on dist-test-slave-wvz7"
server_key: "ea663c2d49a492b2dc9c6151903e24ab"
server_key_iv: "d983b1a1e673a539c014de9ec675b070"
server_key_version: "encryptionkey@0"
I20260619 16:48:42.012135 22920 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887385903558-388-0/minicluster-data/master-2/wal/instance:
uuid: "3bbfc14275c94e54ae7c59dce399387f"
format_stamp: "Formatted at 2026-06-19 16:48:42 on dist-test-slave-wvz7"
server_key: "ea663c2d49a492b2dc9c6151903e24ab"
server_key_iv: "d983b1a1e673a539c014de9ec675b070"
server_key_version: "encryptionkey@0"
I20260619 16:48:42.012609 22913 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 38283 } attempt: 4
I20260619 16:48:42.013785 22915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:48:42.013223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:38283 (local address 127.0.0.1:58160)
0619 16:48:42.013465 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0619 16:48:42.013569 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:38283: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260619 16:48:42.014916 22915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:48:42.014507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:38283 (local address 127.0.0.1:58176)
0619 16:48:42.014693 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0619 16:48:42.014778 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:38283: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260619 16:48:42.015148 22913 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:38283: Network error: Client connection negotiation failed: client connection to 127.0.97.60:38283: connect: Connection refused (error 111)
I20260619 16:48:42.017554 22920 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260619 16:48:42.020992 22935 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:48:42.022430 22920 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 16:48:42.022574 22920 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887385903558-388-0/minicluster-data/master-2/data,/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887385903558-388-0/minicluster-data/master-2/wal
uuid: "3bbfc14275c94e54ae7c59dce399387f"
format_stamp: "Formatted at 2026-06-19 16:48:42 on dist-test-slave-wvz7"
server_key: "ea663c2d49a492b2dc9c6151903e24ab"
server_key_iv: "d983b1a1e673a539c014de9ec675b070"
server_key_version: "encryptionkey@0"
I20260619 16:48:42.022765 22920 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887385903558-388-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887385903558-388-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887385903558-388-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:48:42.060009 22920 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:48:42.064046 22920 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:48:42.064280 22920 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:48:42.080693 22920 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.60:38283
I20260619 16:48:42.080719 22986 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.60:38283 every 8 connection(s)
I20260619 16:48:42.081987 22920 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887385903558-388-0/minicluster-data/master-2/data/info.pb
I20260619 16:48:42.085469 22987 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:48:42.090155 22839 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 38283 } attempt: 3
I20260619 16:48:42.091159 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu as pid 22920
I20260619 16:48:42.091395 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887385903558-388-0/minicluster-data/master-2/wal/instance
I20260619 16:48:42.091882 388 external_mini_cluster.cc:1428] Setting key c04c1607638eb898f6b64b7bba140e81
I20260619 16:48:42.092253 22987 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35423 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 16:48:42 dist-test-slave-wvz7 krb5kdc[22759](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887721, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
Jun 19 16:48:42 dist-test-slave-wvz7 krb5kdc[22759](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887720, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
Jun 19 16:48:42 dist-test-slave-wvz7 krb5kdc[22759](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887721, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 16:48:42.111891 22843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:48:42.096874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46286 (local address 127.0.97.62:35423)
0619 16:48:42.097051 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0619 16:48:42.097057 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:48:42.097839 (+ 782us) server_negotiation.cc:407] Connection header received
0619 16:48:42.098108 (+ 269us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:48:42.098113 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:48:42.098170 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:48:42.098245 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:48:42.099377 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:48:42.100403 (+ 1026us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:48:42.101986 (+ 1583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:48:42.102649 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:48:42.102787 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:48:42.105863 (+ 3076us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:48:42.105870 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:48:42.105875 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:48:42.105912 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:48:42.108482 (+ 2570us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:48:42.109298 (+ 816us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:48:42.109302 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:48:42.109304 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:48:42.109441 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:48:42.109893 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:48:42.109898 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:48:42.109901 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:48:42.110332 (+ 431us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:48:42.110475 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0619 16:48:42.111291 (+ 816us) server_negotiation.cc:299] Negotiation successful
0619 16:48:42.111476 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68,"spinlock_wait_cycles":46848}
I20260619 16:48:42.112166 22842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:48:42.090827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:38283 (local address 127.0.0.1:58182)
0619 16:48:42.091135 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0619 16:48:42.091156 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 16:48:42.091294 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:48:42.095649 (+ 4355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:48:42.095655 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:48:42.096166 (+ 511us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:48:42.096365 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:48:42.096378 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:48:42.098948 (+ 2570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:48:42.098953 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:48:42.100035 (+ 1082us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:48:42.100047 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:48:42.101255 (+ 1208us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:48:42.101262 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:48:42.101443 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:48:42.102076 (+ 633us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:48:42.102102 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:48:42.104867 (+ 2765us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:48:42.109383 (+ 4516us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:48:42.109388 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:48:42.109392 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:48:42.109906 (+ 514us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:48:42.110315 (+ 409us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:48:42.110319 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:48:42.110323 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:48:42.110467 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:48:42.111084 (+ 617us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:48:42.111089 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:48:42.111288 (+ 199us) client_negotiation.cc:769] Sending connection context
0619 16:48:42.111449 (+ 161us) client_negotiation.cc:240] Negotiation successful
0619 16:48:42.111724 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157,"spinlock_wait_cycles":1792}
I20260619 16:48:42.113397 22991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:48:42.092863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58184 (local address 127.0.97.60:38283)
0619 16:48:42.093683 (+ 820us) server_negotiation.cc:206] Beginning negotiation
0619 16:48:42.093688 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:48:42.094126 (+ 438us) server_negotiation.cc:407] Connection header received
0619 16:48:42.094844 (+ 718us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:48:42.094864 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:48:42.095294 (+ 430us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:48:42.095630 (+ 336us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:48:42.097256 (+ 1626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:48:42.098785 (+ 1529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:48:42.100386 (+ 1601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:48:42.101111 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:48:42.101333 (+ 222us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:48:42.106119 (+ 4786us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:48:42.106125 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:48:42.106129 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:48:42.106163 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:48:42.109612 (+ 3449us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:48:42.110245 (+ 633us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:48:42.110250 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:48:42.110254 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:48:42.110381 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:48:42.110773 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:48:42.110777 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:48:42.110779 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:48:42.111048 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:48:42.111304 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0619 16:48:42.112336 (+ 1032us) server_negotiation.cc:299] Negotiation successful
0619 16:48:42.112539 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":219,"server-negotiator.queue_time_us":679,"thread_start_us":485,"threads_started":1}
I20260619 16:48:42.113451 22992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:48:42.096903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35423 (local address 127.0.0.1:46286)
0619 16:48:42.097695 (+ 792us) negotiation.cc:107] Waiting for socket to connect
0619 16:48:42.097762 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0619 16:48:42.097939 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:48:42.098331 (+ 392us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:48:42.098337 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:48:42.098949 (+ 612us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:48:42.099218 (+ 269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:48:42.099239 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:48:42.100560 (+ 1321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:48:42.100564 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:48:42.101867 (+ 1303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:48:42.101881 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:48:42.102876 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:48:42.102882 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:48:42.103051 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:48:42.103629 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:48:42.103650 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:48:42.105656 (+ 2006us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:48:42.108609 (+ 2953us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:48:42.108620 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:48:42.108632 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:48:42.109183 (+ 551us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:48:42.109546 (+ 363us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:48:42.109551 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:48:42.109554 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:48:42.109772 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:48:42.110439 (+ 667us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:48:42.110447 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:48:42.110885 (+ 438us) client_negotiation.cc:769] Sending connection context
0619 16:48:42.112341 (+ 1456us) client_negotiation.cc:240] Negotiation successful
0619 16:48:42.112690 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":578,"mutex_wait_us":41,"thread_start_us":426,"threads_started":1}
I20260619 16:48:42.113487 22989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:48:42.091911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58182 (local address 127.0.97.60:38283)
0619 16:48:42.092745 (+ 834us) server_negotiation.cc:206] Beginning negotiation
0619 16:48:42.092759 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 16:48:42.092816 (+ 57us) server_negotiation.cc:407] Connection header received
0619 16:48:42.094836 (+ 2020us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:48:42.094864 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:48:42.095295 (+ 431us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:48:42.096237 (+ 942us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:48:42.096526 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:48:42.098785 (+ 2259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:48:42.100159 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:48:42.101111 (+ 952us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:48:42.101330 (+ 219us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:48:42.105074 (+ 3744us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:48:42.105084 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:48:42.105102 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:48:42.105137 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:48:42.109215 (+ 4078us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:48:42.110061 (+ 846us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:48:42.110068 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:48:42.110071 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:48:42.110204 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:48:42.110553 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:48:42.110557 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:48:42.110560 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:48:42.110942 (+ 382us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:48:42.111115 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0619 16:48:42.111615 (+ 500us) server_negotiation.cc:299] Negotiation successful
0619 16:48:42.112033 (+ 418us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":269,"server-negotiator.queue_time_us":602,"spinlock_wait_cycles":53632,"thread_start_us":367,"threads_started":1}
I20260619 16:48:42.114874 22987 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40923 } has no permanent_uuid. Determining permanent_uuid...
I20260619 16:48:42.120237 22839 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5b699388e5ad4dcda958440823bf99a8: Bootstrap starting.
Jun 19 16:48:42 dist-test-slave-wvz7 krb5kdc[22759](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887721, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260619 16:48:42.123924 22839 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5b699388e5ad4dcda958440823bf99a8: Neither blocks nor log segments found. Creating new log.
I20260619 16:48:42.125232 22839 log.cc:824] T 00000000000000000000000000000000 P 5b699388e5ad4dcda958440823bf99a8: Log is configured to *not* fsync() on all Append() calls
Jun 19 16:48:42 dist-test-slave-wvz7 krb5kdc[22759](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887720, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 16:48:42.128561 22839 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5b699388e5ad4dcda958440823bf99a8: No bootstrap required, opened a new log
I20260619 16:48:42.128594 22992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:48:42.115546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:40923 (local address 127.0.0.1:44144)
0619 16:48:42.116022 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0619 16:48:42.116039 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 16:48:42.116140 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:48:42.116614 (+ 474us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:48:42.116618 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:48:42.117022 (+ 404us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:48:42.117175 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:48:42.117185 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:48:42.118211 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:48:42.118216 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:48:42.119463 (+ 1247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:48:42.119475 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:48:42.120420 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:48:42.120426 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:48:42.120562 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:48:42.121052 (+ 490us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:48:42.121067 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:48:42.123158 (+ 2091us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:48:42.126144 (+ 2986us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:48:42.126148 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:48:42.126152 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:48:42.126632 (+ 480us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:48:42.127023 (+ 391us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:48:42.127027 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:48:42.127031 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:48:42.127185 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:48:42.127723 (+ 538us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:48:42.127728 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:48:42.127881 (+ 153us) client_negotiation.cc:769] Sending connection context
0619 16:48:42.128029 (+ 148us) client_negotiation.cc:240] Negotiation successful
0619 16:48:42.128280 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":374}
I20260619 16:48:42.128700 22917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:48:42.115604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44144 (local address 127.0.97.61:40923)
0619 16:48:42.115792 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0619 16:48:42.115797 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:48:42.116119 (+ 322us) server_negotiation.cc:407] Connection header received
0619 16:48:42.116264 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:48:42.116270 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:48:42.116352 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:48:42.116445 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:48:42.117316 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:48:42.118086 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:48:42.119602 (+ 1516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:48:42.120248 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:48:42.120381 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:48:42.123332 (+ 2951us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:48:42.123339 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:48:42.123344 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:48:42.123383 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:48:42.126001 (+ 2618us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:48:42.126766 (+ 765us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:48:42.126770 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:48:42.126772 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:48:42.126888 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:48:42.127283 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:48:42.127286 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:48:42.127289 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:48:42.127587 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:48:42.127733 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0619 16:48:42.128135 (+ 402us) server_negotiation.cc:299] Negotiation successful
0619 16:48:42.128364 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260619 16:48:42.129581 22987 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 38283 } has no permanent_uuid. Determining permanent_uuid...
I20260619 16:48:42.133529 22843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:48:42.116631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46298 (local address 127.0.97.62:35423)
0619 16:48:42.116787 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0619 16:48:42.116792 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:48:42.116807 (+ 15us) server_negotiation.cc:407] Connection header received
0619 16:48:42.116847 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:48:42.116851 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:48:42.116912 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:48:42.117016 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:48:42.118379 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:48:42.119464 (+ 1085us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:48:42.122087 (+ 2623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:48:42.122771 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:48:42.122882 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:48:42.127506 (+ 4624us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:48:42.127512 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:48:42.127516 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:48:42.127548 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:48:42.130746 (+ 3198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:48:42.131496 (+ 750us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:48:42.131501 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:48:42.131505 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:48:42.131668 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:48:42.132061 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:48:42.132066 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:48:42.132069 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:48:42.132413 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:48:42.132654 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0619 16:48:42.132947 (+ 293us) server_negotiation.cc:299] Negotiation successful
0619 16:48:42.133147 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
Jun 19 16:48:42 dist-test-slave-wvz7 krb5kdc[22759](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887721, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
I20260619 16:48:42.139197 22839 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5b699388e5ad4dcda958440823bf99a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b699388e5ad4dcda958440823bf99a8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35423 } } peers { permanent_uuid: "f2c48d2a6aa248ed880e12de4b6b18e5" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40923 } } peers { permanent_uuid: "3bbfc14275c94e54ae7c59dce399387f" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 38283 } }
I20260619 16:48:42.139600 22839 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5b699388e5ad4dcda958440823bf99a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b699388e5ad4dcda958440823bf99a8, State: Initialized, Role: FOLLOWER
I20260619 16:48:42.140153 22839 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5b699388e5ad4dcda958440823bf99a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b699388e5ad4dcda958440823bf99a8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35423 } } peers { permanent_uuid: "f2c48d2a6aa248ed880e12de4b6b18e5" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40923 } } peers { permanent_uuid: "3bbfc14275c94e54ae7c59dce399387f" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 38283 } }
I20260619 16:48:42.141234 22994 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5b699388e5ad4dcda958440823bf99a8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b699388e5ad4dcda958440823bf99a8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35423 } } peers { permanent_uuid: "f2c48d2a6aa248ed880e12de4b6b18e5" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40923 } } peers { permanent_uuid: "3bbfc14275c94e54ae7c59dce399387f" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 38283 } } }
I20260619 16:48:42.141664 22994 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5b699388e5ad4dcda958440823bf99a8 [sys.catalog]: This master's current role is: FOLLOWER
Jun 19 16:48:42 dist-test-slave-wvz7 krb5kdc[22759](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887720, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260619 16:48:42.144014 22992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:48:42.130171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:38283 (local address 127.0.0.1:58192)
0619 16:48:42.130372 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0619 16:48:42.130386 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 16:48:42.130476 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:48:42.130711 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:48:42.130714 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:48:42.131055 (+ 341us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:48:42.131179 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:48:42.131187 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:48:42.132259 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:48:42.132264 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:48:42.133533 (+ 1269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:48:42.133547 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:48:42.134702 (+ 1155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:48:42.134708 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:48:42.134932 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:48:42.135528 (+ 596us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:48:42.135550 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:48:42.137955 (+ 2405us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:48:42.141508 (+ 3553us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:48:42.141513 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:48:42.141517 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:48:42.142010 (+ 493us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:48:42.142386 (+ 376us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:48:42.142391 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:48:42.142395 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:48:42.142545 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:48:42.143021 (+ 476us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:48:42.143026 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:48:42.143160 (+ 134us) client_negotiation.cc:769] Sending connection context
0619 16:48:42.143303 (+ 143us) client_negotiation.cc:240] Negotiation successful
0619 16:48:42.143596 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124}
I20260619 16:48:42.144074 22991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:48:42.130196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58192 (local address 127.0.97.60:38283)
0619 16:48:42.130371 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0619 16:48:42.130377 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:48:42.130446 (+ 69us) server_negotiation.cc:407] Connection header received
0619 16:48:42.130556 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:48:42.130561 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:48:42.130616 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:48:42.130684 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:48:42.131302 (+ 618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:48:42.132131 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:48:42.133692 (+ 1561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:48:42.134316 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:48:42.134455 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:48:42.138119 (+ 3664us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:48:42.138124 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:48:42.138129 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:48:42.138161 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:48:42.141390 (+ 3229us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:48:42.142142 (+ 752us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:48:42.142147 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:48:42.142150 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:48:42.142271 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:48:42.142645 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:48:42.142650 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:48:42.142654 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:48:42.142919 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:48:42.143041 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0619 16:48:42.143438 (+ 397us) server_negotiation.cc:299] Negotiation successful
0619 16:48:42.143760 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":90,"server-negotiator.queue_time_us":33}
I20260619 16:48:42.149189 22917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:48:42.135257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44154 (local address 127.0.97.61:40923)
0619 16:48:42.135412 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0619 16:48:42.135417 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:48:42.135499 (+ 82us) server_negotiation.cc:407] Connection header received
0619 16:48:42.135639 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:48:42.135644 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:48:42.135720 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:48:42.135811 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:48:42.137086 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:48:42.137923 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:48:42.139173 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:48:42.139617 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:48:42.139749 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:48:42.144227 (+ 4478us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:48:42.144234 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:48:42.144239 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:48:42.144276 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:48:42.146917 (+ 2641us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:48:42.147484 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:48:42.147489 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:48:42.147491 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:48:42.147613 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:48:42.147941 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:48:42.147946 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:48:42.147949 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:48:42.148186 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:48:42.148438 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0619 16:48:42.148681 (+ 243us) server_negotiation.cc:299] Negotiation successful
0619 16:48:42.148888 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260619 16:48:42.150048 22987 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3bbfc14275c94e54ae7c59dce399387f: Bootstrap starting.
I20260619 16:48:42.153308 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 16:48:42.153860 22987 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3bbfc14275c94e54ae7c59dce399387f: Neither blocks nor log segments found. Creating new log.
I20260619 16:48:42.155074 22987 log.cc:824] T 00000000000000000000000000000000 P 3bbfc14275c94e54ae7c59dce399387f: Log is configured to *not* fsync() on all Append() calls
I20260619 16:48:42.155747 22814 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "5b699388e5ad4dcda958440823bf99a8"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46298
I20260619 16:48:42.156826 22839 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5b699388e5ad4dcda958440823bf99a8 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:48:42.156884 22814 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5b699388e5ad4dcda958440823bf99a8 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260619 16:48:42.157063 22814 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5b699388e5ad4dcda958440823bf99a8 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:48:42.158330 22987 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3bbfc14275c94e54ae7c59dce399387f: No bootstrap required, opened a new log
I20260619 16:48:42.159736 22814 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5b699388e5ad4dcda958440823bf99a8 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b699388e5ad4dcda958440823bf99a8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35423 } } peers { permanent_uuid: "f2c48d2a6aa248ed880e12de4b6b18e5" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40923 } } peers { permanent_uuid: "3bbfc14275c94e54ae7c59dce399387f" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 38283 } }
I20260619 16:48:42.160820 22814 leader_election.cc:288] T 00000000000000000000000000000000 P 5b699388e5ad4dcda958440823bf99a8 [CANDIDATE]: Term 1 election: Requested vote from peers f2c48d2a6aa248ed880e12de4b6b18e5 (127.0.97.61:40923), 3bbfc14275c94e54ae7c59dce399387f (127.0.97.60:38283)
I20260619 16:48:42.169117 22987 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3bbfc14275c94e54ae7c59dce399387f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b699388e5ad4dcda958440823bf99a8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35423 } } peers { permanent_uuid: "f2c48d2a6aa248ed880e12de4b6b18e5" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40923 } } peers { permanent_uuid: "3bbfc14275c94e54ae7c59dce399387f" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 38283 } }
I20260619 16:48:42.169579 22987 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3bbfc14275c94e54ae7c59dce399387f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3bbfc14275c94e54ae7c59dce399387f, State: Initialized, Role: FOLLOWER
I20260619 16:48:42.170152 22987 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3bbfc14275c94e54ae7c59dce399387f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b699388e5ad4dcda958440823bf99a8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35423 } } peers { permanent_uuid: "f2c48d2a6aa248ed880e12de4b6b18e5" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40923 } } peers { permanent_uuid: "3bbfc14275c94e54ae7c59dce399387f" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 38283 } }
I20260619 16:48:42.171351 23006 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3bbfc14275c94e54ae7c59dce399387f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b699388e5ad4dcda958440823bf99a8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35423 } } peers { permanent_uuid: "f2c48d2a6aa248ed880e12de4b6b18e5" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40923 } } peers { permanent_uuid: "3bbfc14275c94e54ae7c59dce399387f" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 38283 } } }
I20260619 16:48:42.172258 23006 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3bbfc14275c94e54ae7c59dce399387f [sys.catalog]: This master's current role is: FOLLOWER
I20260619 16:48:42.172485 22987 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3bbfc14275c94e54ae7c59dce399387f [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:48:42.172541 22962 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5b699388e5ad4dcda958440823bf99a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "3bbfc14275c94e54ae7c59dce399387f"
I20260619 16:48:42.172969 22962 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3bbfc14275c94e54ae7c59dce399387f [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:48:42.176633 22962 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 3bbfc14275c94e54ae7c59dce399387f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5b699388e5ad4dcda958440823bf99a8 in term 1.
I20260619 16:48:42.177529 22791 leader_election.cc:302] T 00000000000000000000000000000000 P 5b699388e5ad4dcda958440823bf99a8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3bbfc14275c94e54ae7c59dce399387f, 5b699388e5ad4dcda958440823bf99a8; no voters:
I20260619 16:48:42.178220 22994 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5b699388e5ad4dcda958440823bf99a8 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:48:42.178797 22994 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5b699388e5ad4dcda958440823bf99a8 [term 1 LEADER]: Becoming Leader. State: Replica: 5b699388e5ad4dcda958440823bf99a8, State: Running, Role: LEADER
I20260619 16:48:42.179354 22994 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5b699388e5ad4dcda958440823bf99a8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b699388e5ad4dcda958440823bf99a8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35423 } } peers { permanent_uuid: "f2c48d2a6aa248ed880e12de4b6b18e5" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40923 } } peers { permanent_uuid: "3bbfc14275c94e54ae7c59dce399387f" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 38283 } }
I20260619 16:48:42.184492 23012 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5b699388e5ad4dcda958440823bf99a8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5b699388e5ad4dcda958440823bf99a8. Latest consensus state: current_term: 1 leader_uuid: "5b699388e5ad4dcda958440823bf99a8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b699388e5ad4dcda958440823bf99a8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35423 } } peers { permanent_uuid: "f2c48d2a6aa248ed880e12de4b6b18e5" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40923 } } peers { permanent_uuid: "3bbfc14275c94e54ae7c59dce399387f" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 38283 } } }
I20260619 16:48:42.184916 23012 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5b699388e5ad4dcda958440823bf99a8 [sys.catalog]: This master's current role is: LEADER
W20260619 16:48:42.196877 23015 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5b699388e5ad4dcda958440823bf99a8: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 16:48:42.197062 23015 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 16:48:42.197252 23016 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:48:42.199378 23016 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:48:42.189519 22772 master_runner.cc:186] Error getting master registration for 127.0.97.61:40923: OK, instance_id { permanent_uuid: "f2c48d2a6aa248ed880e12de4b6b18e5" instance_seqno: 1781887721595865 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260619 16:48:42.201349 22772 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260619 16:48:42.220657 22962 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 3bbfc14275c94e54ae7c59dce399387f [term 1 FOLLOWER]: Refusing update from remote peer 5b699388e5ad4dcda958440823bf99a8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:48:42.221704 23012 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 5b699388e5ad4dcda958440823bf99a8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3bbfc14275c94e54ae7c59dce399387f" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 38283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:48:42.221863 23029 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260619 16:48:42.226982 22843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:48:42.204802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46310 (local address 127.0.97.62:35423)
0619 16:48:42.204964 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0619 16:48:42.204970 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:48:42.205307 (+ 337us) server_negotiation.cc:407] Connection header received
0619 16:48:42.205719 (+ 412us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:48:42.205725 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:48:42.205827 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:48:42.205900 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:48:42.209442 (+ 3542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:48:42.210655 (+ 1213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:48:42.213850 (+ 3195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:48:42.214615 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:48:42.214716 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:48:42.220234 (+ 5518us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:48:42.220242 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:48:42.220247 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:48:42.220290 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:48:42.223996 (+ 3706us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:48:42.224760 (+ 764us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:48:42.224765 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:48:42.224768 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:48:42.224925 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:48:42.225338 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:48:42.225343 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:48:42.225345 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:48:42.225653 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:48:42.225901 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0619 16:48:42.226396 (+ 495us) server_negotiation.cc:299] Negotiation successful
0619 16:48:42.226632 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260619 16:48:42.239257 23006 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3bbfc14275c94e54ae7c59dce399387f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5b699388e5ad4dcda958440823bf99a8. Latest consensus state: current_term: 1 leader_uuid: "5b699388e5ad4dcda958440823bf99a8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b699388e5ad4dcda958440823bf99a8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35423 } } peers { permanent_uuid: "f2c48d2a6aa248ed880e12de4b6b18e5" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40923 } } peers { permanent_uuid: "3bbfc14275c94e54ae7c59dce399387f" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 38283 } } }
I20260619 16:48:42.239804 23006 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3bbfc14275c94e54ae7c59dce399387f [sys.catalog]: This master's current role is: FOLLOWER
I20260619 16:48:42.232352 22920 master_runner.cc:186] Error getting master registration for 127.0.97.61:40923: OK, instance_id { permanent_uuid: "f2c48d2a6aa248ed880e12de4b6b18e5" instance_seqno: 1781887721595865 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260619 16:48:42.244220 22920 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260619 16:48:42.244510 22991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:48:42.218347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58208 (local address 127.0.97.60:38283)
0619 16:48:42.218558 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0619 16:48:42.218564 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:48:42.227828 (+ 9264us) server_negotiation.cc:407] Connection header received
0619 16:48:42.228036 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:48:42.228041 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:48:42.228112 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:48:42.228178 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:48:42.229872 (+ 1694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:48:42.230893 (+ 1021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:48:42.232510 (+ 1617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:48:42.233209 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:48:42.233301 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:48:42.236392 (+ 3091us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:48:42.236399 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:48:42.236403 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:48:42.236442 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:48:42.240243 (+ 3801us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:48:42.241606 (+ 1363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:48:42.241612 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:48:42.241616 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:48:42.241808 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:48:42.242455 (+ 647us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:48:42.242460 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:48:42.242464 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:48:42.242835 (+ 371us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:48:42.243090 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0619 16:48:42.243983 (+ 893us) server_negotiation.cc:299] Negotiation successful
0619 16:48:42.244226 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260619 16:48:42.245551 23006 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3bbfc14275c94e54ae7c59dce399387f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5b699388e5ad4dcda958440823bf99a8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b699388e5ad4dcda958440823bf99a8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35423 } } peers { permanent_uuid: "f2c48d2a6aa248ed880e12de4b6b18e5" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40923 } } peers { permanent_uuid: "3bbfc14275c94e54ae7c59dce399387f" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 38283 } } }
I20260619 16:48:42.245816 23006 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3bbfc14275c94e54ae7c59dce399387f [sys.catalog]: This master's current role is: FOLLOWER
I20260619 16:48:42.245990 23031 mvcc.cc:205] Tried to move back new op lower bound from 7298612110186356736 to 7298612110051819520. Current Snapshot: MvccSnapshot[applied={T|T < 7298612110186356736}]
I20260619 16:48:42.248052 23025 mvcc.cc:205] Tried to move back new op lower bound from 7298612110186356736 to 7298612110051819520. Current Snapshot: MvccSnapshot[applied={T|T < 7298612110186356736}]
I20260619 16:48:42.260255 22917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:48:42.205688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44170 (local address 127.0.97.61:40923)
0619 16:48:42.205862 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0619 16:48:42.205867 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:48:42.214296 (+ 8429us) server_negotiation.cc:407] Connection header received
0619 16:48:42.214539 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:48:42.214545 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:48:42.214625 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:48:42.214693 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:48:42.233095 (+ 18402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:48:42.234172 (+ 1077us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:48:42.237870 (+ 3698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:48:42.238515 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:48:42.238638 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:48:42.253881 (+ 15243us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:48:42.253888 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:48:42.253893 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:48:42.253942 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:48:42.257505 (+ 3563us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:48:42.258330 (+ 825us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:48:42.258337 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:48:42.258341 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:48:42.258474 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:48:42.258864 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:48:42.258870 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:48:42.258874 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:48:42.259178 (+ 304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:48:42.259407 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0619 16:48:42.259694 (+ 287us) server_negotiation.cc:299] Negotiation successful
0619 16:48:42.259915 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260619 16:48:42.260288 23012 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5b699388e5ad4dcda958440823bf99a8 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5b699388e5ad4dcda958440823bf99a8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b699388e5ad4dcda958440823bf99a8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35423 } } peers { permanent_uuid: "f2c48d2a6aa248ed880e12de4b6b18e5" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40923 } } peers { permanent_uuid: "3bbfc14275c94e54ae7c59dce399387f" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 38283 } } }
I20260619 16:48:42.260563 23012 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5b699388e5ad4dcda958440823bf99a8 [sys.catalog]: This master's current role is: LEADER
I20260619 16:48:42.261329 23016 catalog_manager.cc:1347] Generated new cluster ID: 9aef717e17ba411ab840bcaa6743c61b
I20260619 16:48:42.261438 23016 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:48:42.282086 23016 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 16:48:42.283942 23016 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:48:42.296990 23016 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5b699388e5ad4dcda958440823bf99a8: Generated new TSK 0
I20260619 16:48:42.298221 23016 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 16:48:42.314350 22913 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 38283 } attempt: 5
Jun 19 16:48:42 dist-test-slave-wvz7 krb5kdc[22759](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887721, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
I20260619 16:48:42.329506 22991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:48:42.314994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58214 (local address 127.0.97.60:38283)
0619 16:48:42.315160 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0619 16:48:42.315165 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:48:42.315185 (+ 20us) server_negotiation.cc:407] Connection header received
0619 16:48:42.315328 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:48:42.315333 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:48:42.315402 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:48:42.315481 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:48:42.316429 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:48:42.317446 (+ 1017us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:48:42.319044 (+ 1598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:48:42.319661 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:48:42.319774 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:48:42.323597 (+ 3823us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:48:42.323603 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:48:42.323606 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:48:42.323636 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:48:42.326786 (+ 3150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:48:42.327559 (+ 773us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:48:42.327564 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:48:42.327568 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:48:42.327723 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:48:42.328114 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:48:42.328120 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:48:42.328123 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:48:42.328461 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:48:42.328605 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0619 16:48:42.328970 (+ 365us) server_negotiation.cc:299] Negotiation successful
0619 16:48:42.329166 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260619 16:48:42.329521 22915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:48:42.314843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:38283 (local address 127.0.0.1:58214)
0619 16:48:42.315064 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0619 16:48:42.315080 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 16:48:42.315185 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:48:42.315542 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:48:42.315548 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:48:42.316056 (+ 508us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:48:42.316285 (+ 229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:48:42.316298 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:48:42.317603 (+ 1305us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:48:42.317608 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:48:42.318879 (+ 1271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:48:42.318895 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:48:42.319977 (+ 1082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:48:42.319983 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:48:42.320195 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:48:42.320842 (+ 647us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:48:42.320864 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:48:42.323419 (+ 2555us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:48:42.326902 (+ 3483us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:48:42.326907 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:48:42.326912 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:48:42.327461 (+ 549us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:48:42.327841 (+ 380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:48:42.327846 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:48:42.327850 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:48:42.328010 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:48:42.328568 (+ 558us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:48:42.328573 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:48:42.328726 (+ 153us) client_negotiation.cc:769] Sending connection context
0619 16:48:42.328869 (+ 143us) client_negotiation.cc:240] Negotiation successful
0619 16:48:42.329147 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117}
I20260619 16:48:42.335255 22913 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f2c48d2a6aa248ed880e12de4b6b18e5: Bootstrap starting.
I20260619 16:48:42.339318 22913 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f2c48d2a6aa248ed880e12de4b6b18e5: Neither blocks nor log segments found. Creating new log.
I20260619 16:48:42.340571 22913 log.cc:824] T 00000000000000000000000000000000 P f2c48d2a6aa248ed880e12de4b6b18e5: Log is configured to *not* fsync() on all Append() calls
I20260619 16:48:42.348081 22913 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f2c48d2a6aa248ed880e12de4b6b18e5: No bootstrap required, opened a new log
I20260619 16:48:42.360311 22913 raft_consensus.cc:348] T 00000000000000000000000000000000 P f2c48d2a6aa248ed880e12de4b6b18e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b699388e5ad4dcda958440823bf99a8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35423 } } peers { permanent_uuid: "f2c48d2a6aa248ed880e12de4b6b18e5" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40923 } } peers { permanent_uuid: "3bbfc14275c94e54ae7c59dce399387f" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 38283 } }
I20260619 16:48:42.360700 22913 raft_consensus.cc:729] T 00000000000000000000000000000000 P f2c48d2a6aa248ed880e12de4b6b18e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f2c48d2a6aa248ed880e12de4b6b18e5, State: Initialized, Role: FOLLOWER
I20260619 16:48:42.361241 22913 consensus_queue.cc:260] T 00000000000000000000000000000000 P f2c48d2a6aa248ed880e12de4b6b18e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b699388e5ad4dcda958440823bf99a8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35423 } } peers { permanent_uuid: "f2c48d2a6aa248ed880e12de4b6b18e5" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40923 } } peers { permanent_uuid: "3bbfc14275c94e54ae7c59dce399387f" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 38283 } }
I20260619 16:48:42.362584 23038 sys_catalog.cc:455] T 00000000000000000000000000000000 P f2c48d2a6aa248ed880e12de4b6b18e5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b699388e5ad4dcda958440823bf99a8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35423 } } peers { permanent_uuid: "f2c48d2a6aa248ed880e12de4b6b18e5" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40923 } } peers { permanent_uuid: "3bbfc14275c94e54ae7c59dce399387f" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 38283 } } }
I20260619 16:48:42.363318 23038 sys_catalog.cc:458] T 00000000000000000000000000000000 P f2c48d2a6aa248ed880e12de4b6b18e5 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 16:48:42.363498 22913 sys_catalog.cc:564] T 00000000000000000000000000000000 P f2c48d2a6aa248ed880e12de4b6b18e5 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:48:42.363860 22887 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f2c48d2a6aa248ed880e12de4b6b18e5 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:48:42.367482 22887 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f2c48d2a6aa248ed880e12de4b6b18e5 [term 1 FOLLOWER]: Refusing update from remote peer 5b699388e5ad4dcda958440823bf99a8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:48:42.368510 23012 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 5b699388e5ad4dcda958440823bf99a8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f2c48d2a6aa248ed880e12de4b6b18e5" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40923 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:48:42.369202 22888 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5b699388e5ad4dcda958440823bf99a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "f2c48d2a6aa248ed880e12de4b6b18e5"
I20260619 16:48:42.373147 22888 raft_consensus.cc:2454] T 00000000000000000000000000000000 P f2c48d2a6aa248ed880e12de4b6b18e5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5b699388e5ad4dcda958440823bf99a8 in term 1.
W20260619 16:48:42.423322 22917 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:48:42.413293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44186 (local address 127.0.97.61:40923)
0619 16:48:42.413506 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0619 16:48:42.413513 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:48:42.419237 (+ 5724us) server_negotiation.cc:407] Connection header received
0619 16:48:42.420899 (+ 1662us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:48:42.420907 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:48:42.421026 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:48:42.421101 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:48:42.423054 (+ 1953us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44186: BlockingRecv error: recv got EOF from 127.0.0.1:44186 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
W20260619 16:48:42.425192 22843 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:48:42.404838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46320 (local address 127.0.97.62:35423)
0619 16:48:42.405070 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0619 16:48:42.405077 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:48:42.418737 (+ 13660us) server_negotiation.cc:407] Connection header received
0619 16:48:42.421609 (+ 2872us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:48:42.421624 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:48:42.421714 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:48:42.421821 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:48:42.424931 (+ 3110us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46320: BlockingRecv error: recv got EOF from 127.0.0.1:46320 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
I20260619 16:48:42.427433 23038 sys_catalog.cc:455] T 00000000000000000000000000000000 P f2c48d2a6aa248ed880e12de4b6b18e5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5b699388e5ad4dcda958440823bf99a8. Latest consensus state: current_term: 1 leader_uuid: "5b699388e5ad4dcda958440823bf99a8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b699388e5ad4dcda958440823bf99a8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35423 } } peers { permanent_uuid: "f2c48d2a6aa248ed880e12de4b6b18e5" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40923 } } peers { permanent_uuid: "3bbfc14275c94e54ae7c59dce399387f" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 38283 } } }
I20260619 16:48:42.427749 23038 sys_catalog.cc:458] T 00000000000000000000000000000000 P f2c48d2a6aa248ed880e12de4b6b18e5 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 16:48:42.428504 23040 sys_catalog.cc:455] T 00000000000000000000000000000000 P f2c48d2a6aa248ed880e12de4b6b18e5 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5b699388e5ad4dcda958440823bf99a8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b699388e5ad4dcda958440823bf99a8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35423 } } peers { permanent_uuid: "f2c48d2a6aa248ed880e12de4b6b18e5" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40923 } } peers { permanent_uuid: "3bbfc14275c94e54ae7c59dce399387f" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 38283 } } }
I20260619 16:48:42.428869 23040 sys_catalog.cc:458] T 00000000000000000000000000000000 P f2c48d2a6aa248ed880e12de4b6b18e5 [sys.catalog]: This master's current role is: FOLLOWER
W20260619 16:48:42.429936 22991 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:48:42.419820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58226 (local address 127.0.97.60:38283)
0619 16:48:42.419999 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0619 16:48:42.420005 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:48:42.428112 (+ 8107us) server_negotiation.cc:407] Connection header received
0619 16:48:42.428448 (+ 336us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:48:42.428455 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:48:42.428555 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:48:42.428633 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:48:42.429678 (+ 1045us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58226: BlockingRecv error: recv got EOF from 127.0.0.1:58226 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
I20260619 16:48:42.435307 23061 catalog_manager.cc:1259] Loaded cluster ID: 9aef717e17ba411ab840bcaa6743c61b
I20260619 16:48:42.435392 23061 catalog_manager.cc:1552] T 00000000000000000000000000000000 P f2c48d2a6aa248ed880e12de4b6b18e5: loading cluster ID for follower catalog manager: success
I20260619 16:48:42.439111 23061 catalog_manager.cc:1574] T 00000000000000000000000000000000 P f2c48d2a6aa248ed880e12de4b6b18e5: acquiring CA information for follower catalog manager: success
I20260619 16:48:42.441699 23061 catalog_manager.cc:1602] T 00000000000000000000000000000000 P f2c48d2a6aa248ed880e12de4b6b18e5: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 16:48:42.451725 22917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:48:42.444226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44194 (local address 127.0.97.61:40923)
0619 16:48:42.444388 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0619 16:48:42.444394 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:48:42.444838 (+ 444us) server_negotiation.cc:407] Connection header received
0619 16:48:42.445062 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:48:42.445068 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:48:42.445268 (+ 200us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:48:42.445423 (+ 155us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 16:48:42.445896 (+ 473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:48:42.447009 (+ 1113us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:48:42.448979 (+ 1970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:48:42.449670 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:48:42.449840 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:48:42.450393 (+ 553us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 16:48:42.451063 (+ 670us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 16:48:42.451225 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0619 16:48:42.451310 (+ 85us) server_negotiation.cc:299] Negotiation successful
0619 16:48:42.451402 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
W20260619 16:48:42.463932 22991 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:48:42.460186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58236 (local address 127.0.97.60:38283)
0619 16:48:42.460363 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0619 16:48:42.460369 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:48:42.462355 (+ 1986us) server_negotiation.cc:407] Connection header received
0619 16:48:42.462606 (+ 251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:48:42.462612 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:48:42.462753 (+ 141us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:48:42.462863 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:48:42.463709 (+ 846us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58236: BlockingRecv error: recv got EOF from 127.0.0.1:58236 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
I20260619 16:48:42.484382 22917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:48:42.475493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44200 (local address 127.0.97.61:40923)
0619 16:48:42.475695 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0619 16:48:42.475701 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:48:42.478293 (+ 2592us) server_negotiation.cc:407] Connection header received
0619 16:48:42.478504 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:48:42.478510 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:48:42.478597 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:48:42.478706 (+ 109us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 16:48:42.479168 (+ 462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:48:42.480312 (+ 1144us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:48:42.482254 (+ 1942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:48:42.482934 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:48:42.483057 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:48:42.483542 (+ 485us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 16:48:42.483697 (+ 155us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 16:48:42.483763 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0619 16:48:42.483995 (+ 232us) server_negotiation.cc:299] Negotiation successful
0619 16:48:42.484080 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
W20260619 16:48:42.499846 22991 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:48:42.495692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58248 (local address 127.0.97.60:38283)
0619 16:48:42.495864 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0619 16:48:42.495870 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:48:42.498316 (+ 2446us) server_negotiation.cc:407] Connection header received
0619 16:48:42.498511 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:48:42.498517 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:48:42.498613 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:48:42.498721 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:48:42.499595 (+ 874us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58248: BlockingRecv error: recv got EOF from 127.0.0.1:58248 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
I20260619 16:48:42.511520 22917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:48:42.504733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44216 (local address 127.0.97.61:40923)
0619 16:48:42.504900 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0619 16:48:42.504906 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:48:42.505387 (+ 481us) server_negotiation.cc:407] Connection header received
0619 16:48:42.505567 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:48:42.505573 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:48:42.505679 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:48:42.505826 (+ 147us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 16:48:42.506455 (+ 629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:48:42.507582 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:48:42.509440 (+ 1858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:48:42.510163 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:48:42.510282 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:48:42.510720 (+ 438us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 16:48:42.510880 (+ 160us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 16:48:42.510965 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0619 16:48:42.511125 (+ 160us) server_negotiation.cc:299] Negotiation successful
0619 16:48:42.511209 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
W20260619 16:48:42.520093 22991 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:48:42.518123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58262 (local address 127.0.97.60:38283)
0619 16:48:42.518283 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0619 16:48:42.518288 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:48:42.518747 (+ 459us) server_negotiation.cc:407] Connection header received
0619 16:48:42.518906 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:48:42.518911 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:48:42.518999 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:48:42.519104 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:48:42.519886 (+ 782us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58262: BlockingRecv error: recv got EOF from 127.0.0.1:58262 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
I20260619 16:48:42.534195 22917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:48:42.527543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44218 (local address 127.0.97.61:40923)
0619 16:48:42.527759 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0619 16:48:42.527765 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:48:42.528319 (+ 554us) server_negotiation.cc:407] Connection header received
0619 16:48:42.528480 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:48:42.528485 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:48:42.528569 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:48:42.528666 (+ 97us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 16:48:42.529075 (+ 409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:48:42.530235 (+ 1160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:48:42.532119 (+ 1884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:48:42.532822 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:48:42.532942 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:48:42.533418 (+ 476us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 16:48:42.533549 (+ 131us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 16:48:42.533631 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0619 16:48:42.533815 (+ 184us) server_negotiation.cc:299] Negotiation successful
0619 16:48:42.533910 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98}
W20260619 16:48:42.547785 22991 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:48:42.540895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58272 (local address 127.0.97.60:38283)
0619 16:48:42.541176 (+ 281us) server_negotiation.cc:206] Beginning negotiation
0619 16:48:42.541182 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:48:42.546308 (+ 5126us) server_negotiation.cc:407] Connection header received
0619 16:48:42.546478 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:48:42.546484 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:48:42.546584 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:48:42.546688 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:48:42.547547 (+ 859us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58272: BlockingRecv error: recv got EOF from 127.0.0.1:58272 (error 108)
Metrics: {"server-negotiator.queue_time_us":151}
I20260619 16:48:42.566854 22917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:48:42.558823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44220 (local address 127.0.97.61:40923)
0619 16:48:42.559001 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0619 16:48:42.559007 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:48:42.559523 (+ 516us) server_negotiation.cc:407] Connection header received
0619 16:48:42.559706 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:48:42.559711 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:48:42.559801 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:48:42.559911 (+ 110us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 16:48:42.560351 (+ 440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:48:42.561487 (+ 1136us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:48:42.563440 (+ 1953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:48:42.564181 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:48:42.564305 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:48:42.564826 (+ 521us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 16:48:42.564991 (+ 165us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 16:48:42.565089 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0619 16:48:42.566422 (+ 1333us) server_negotiation.cc:299] Negotiation successful
0619 16:48:42.566528 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
W20260619 16:48:42.576449 22991 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:48:42.574391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58278 (local address 127.0.97.60:38283)
0619 16:48:42.574581 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0619 16:48:42.574587 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:48:42.575072 (+ 485us) server_negotiation.cc:407] Connection header received
0619 16:48:42.575272 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:48:42.575278 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:48:42.575372 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:48:42.575468 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:48:42.576229 (+ 761us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58278: BlockingRecv error: recv got EOF from 127.0.0.1:58278 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
I20260619 16:48:42.604188 22917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:48:42.596654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44234 (local address 127.0.97.61:40923)
0619 16:48:42.596878 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0619 16:48:42.596884 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:48:42.597511 (+ 627us) server_negotiation.cc:407] Connection header received
0619 16:48:42.597868 (+ 357us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:48:42.597875 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:48:42.597969 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:48:42.598064 (+ 95us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 16:48:42.598608 (+ 544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:48:42.599819 (+ 1211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:48:42.601820 (+ 2001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:48:42.602517 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:48:42.602610 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:48:42.603284 (+ 674us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 16:48:42.603471 (+ 187us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 16:48:42.603539 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0619 16:48:42.603810 (+ 271us) server_negotiation.cc:299] Negotiation successful
0619 16:48:42.603898 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102}
W20260619 16:48:42.633222 22991 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:48:42.630911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58290 (local address 127.0.97.60:38283)
0619 16:48:42.631121 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0619 16:48:42.631136 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 16:48:42.631660 (+ 524us) server_negotiation.cc:407] Connection header received
0619 16:48:42.631862 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:48:42.631868 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:48:42.631969 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:48:42.632077 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:48:42.632961 (+ 884us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58290: BlockingRecv error: recv got EOF from 127.0.0.1:58290 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
I20260619 16:48:42.677630 22917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:48:42.671229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44248 (local address 127.0.97.61:40923)
0619 16:48:42.671420 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0619 16:48:42.671427 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:48:42.671922 (+ 495us) server_negotiation.cc:407] Connection header received
0619 16:48:42.672104 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:48:42.672109 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:48:42.672269 (+ 160us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:48:42.672375 (+ 106us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 16:48:42.672843 (+ 468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:48:42.673980 (+ 1137us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:48:42.675635 (+ 1655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:48:42.676343 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:48:42.676463 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:48:42.676887 (+ 424us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 16:48:42.677014 (+ 127us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 16:48:42.677100 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0619 16:48:42.677225 (+ 125us) server_negotiation.cc:299] Negotiation successful
0619 16:48:42.677306 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
W20260619 16:48:42.685276 22991 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:48:42.683708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58298 (local address 127.0.97.60:38283)
0619 16:48:42.683895 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0619 16:48:42.683900 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:48:42.684080 (+ 180us) server_negotiation.cc:407] Connection header received
0619 16:48:42.684223 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:48:42.684228 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:48:42.684333 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:48:42.684438 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:48:42.685058 (+ 620us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58298: BlockingRecv error: recv got EOF from 127.0.0.1:58298 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
I20260619 16:48:42.758324 22917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:48:42.752373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44264 (local address 127.0.97.61:40923)
0619 16:48:42.752573 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0619 16:48:42.752579 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:48:42.752855 (+ 276us) server_negotiation.cc:407] Connection header received
0619 16:48:42.753022 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:48:42.753028 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:48:42.753128 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:48:42.753276 (+ 148us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 16:48:42.753803 (+ 527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:48:42.754956 (+ 1153us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:48:42.756294 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:48:42.757014 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:48:42.757146 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:48:42.757523 (+ 377us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 16:48:42.757681 (+ 158us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 16:48:42.757819 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0619 16:48:42.757938 (+ 119us) server_negotiation.cc:299] Negotiation successful
0619 16:48:42.758020 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
W20260619 16:48:42.765794 22991 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:48:42.763974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58304 (local address 127.0.97.60:38283)
0619 16:48:42.764213 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0619 16:48:42.764219 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:48:42.764428 (+ 209us) server_negotiation.cc:407] Connection header received
0619 16:48:42.764600 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:48:42.764606 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:48:42.764699 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:48:42.764810 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:48:42.765524 (+ 714us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58304: BlockingRecv error: recv got EOF from 127.0.0.1:58304 (error 108)
Metrics: {"server-negotiator.queue_time_us":94}
I20260619 16:48:42.901676 22917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:48:42.896814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37516 (local address 127.0.97.61:40923)
0619 16:48:42.897006 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0619 16:48:42.897011 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:48:42.897318 (+ 307us) server_negotiation.cc:407] Connection header received
0619 16:48:42.897521 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:48:42.897525 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:48:42.897611 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:48:42.897691 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 16:48:42.898146 (+ 455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:48:42.898818 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:48:42.900103 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:48:42.900510 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:48:42.900629 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:48:42.900997 (+ 368us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 16:48:42.901117 (+ 120us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 16:48:42.901190 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0619 16:48:42.901335 (+ 145us) server_negotiation.cc:299] Negotiation successful
0619 16:48:42.901411 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
W20260619 16:48:42.912276 22991 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:48:42.910595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50628 (local address 127.0.97.60:38283)
0619 16:48:42.910802 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0619 16:48:42.910808 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:48:42.911072 (+ 264us) server_negotiation.cc:407] Connection header received
0619 16:48:42.911264 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:48:42.911268 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:48:42.911340 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:48:42.911423 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:48:42.912062 (+ 639us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50628: BlockingRecv error: recv got EOF from 127.0.0.1:50628 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
I20260619 16:48:42.978317 23012 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5b699388e5ad4dcda958440823bf99a8 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5b699388e5ad4dcda958440823bf99a8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b699388e5ad4dcda958440823bf99a8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35423 } } peers { permanent_uuid: "f2c48d2a6aa248ed880e12de4b6b18e5" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40923 } } peers { permanent_uuid: "3bbfc14275c94e54ae7c59dce399387f" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 38283 } } }
I20260619 16:48:42.978643 23012 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5b699388e5ad4dcda958440823bf99a8 [sys.catalog]: This master's current role is: LEADER
I20260619 16:48:43.178689 22917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:48:43.173951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37520 (local address 127.0.97.61:40923)
0619 16:48:43.174170 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0619 16:48:43.174177 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:48:43.174353 (+ 176us) server_negotiation.cc:407] Connection header received
0619 16:48:43.174556 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:48:43.174560 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:48:43.174640 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:48:43.174736 (+ 96us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 16:48:43.175178 (+ 442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:48:43.175867 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:48:43.177148 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:48:43.177524 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:48:43.177633 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:48:43.178039 (+ 406us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 16:48:43.178151 (+ 112us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 16:48:43.178237 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0619 16:48:43.178375 (+ 138us) server_negotiation.cc:299] Negotiation successful
0619 16:48:43.178453 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
W20260619 16:48:43.185439 22991 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:48:43.183793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50634 (local address 127.0.97.60:38283)
0619 16:48:43.183984 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0619 16:48:43.183990 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:48:43.184125 (+ 135us) server_negotiation.cc:407] Connection header received
0619 16:48:43.184302 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:48:43.184307 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:48:43.184378 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:48:43.184458 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:48:43.185190 (+ 732us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50634: BlockingRecv error: recv got EOF from 127.0.0.1:50634 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
I20260619 16:48:43.226018 23029 catalog_manager.cc:1259] Loaded cluster ID: 9aef717e17ba411ab840bcaa6743c61b
I20260619 16:48:43.226141 23029 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 3bbfc14275c94e54ae7c59dce399387f: loading cluster ID for follower catalog manager: success
I20260619 16:48:43.228672 23029 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 3bbfc14275c94e54ae7c59dce399387f: acquiring CA information for follower catalog manager: success
I20260619 16:48:43.230422 23029 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 3bbfc14275c94e54ae7c59dce399387f: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 16:48:43.705334 23172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:48:43.700402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37526 (local address 127.0.97.61:40923)
0619 16:48:43.700877 (+ 475us) server_negotiation.cc:206] Beginning negotiation
0619 16:48:43.700883 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:48:43.700977 (+ 94us) server_negotiation.cc:407] Connection header received
0619 16:48:43.701178 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:48:43.701183 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:48:43.701253 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:48:43.701340 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 16:48:43.701725 (+ 385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:48:43.702455 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:48:43.703754 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:48:43.704205 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:48:43.704318 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:48:43.704683 (+ 365us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 16:48:43.704784 (+ 101us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 16:48:43.704856 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0619 16:48:43.704982 (+ 126us) server_negotiation.cc:299] Negotiation successful
0619 16:48:43.705058 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":365,"thread_start_us":247,"threads_started":1}
I20260619 16:48:43.715698 23178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:48:43.710420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50644 (local address 127.0.97.60:38283)
0619 16:48:43.710885 (+ 465us) server_negotiation.cc:206] Beginning negotiation
0619 16:48:43.710891 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:48:43.710910 (+ 19us) server_negotiation.cc:407] Connection header received
0619 16:48:43.711035 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:48:43.711039 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:48:43.711186 (+ 147us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:48:43.711292 (+ 106us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 16:48:43.711660 (+ 368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:48:43.712355 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:48:43.713589 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:48:43.714050 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:48:43.714158 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:48:43.714609 (+ 451us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 16:48:43.715151 (+ 542us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 16:48:43.715299 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0619 16:48:43.715351 (+ 52us) server_negotiation.cc:299] Negotiation successful
0619 16:48:43.715429 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":339,"thread_start_us":222,"threads_started":1}
I20260619 16:48:43.731455 23172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:48:43.721767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37528 (local address 127.0.97.61:40923)
0619 16:48:43.721950 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0619 16:48:43.721956 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:48:43.725412 (+ 3456us) server_negotiation.cc:407] Connection header received
0619 16:48:43.725687 (+ 275us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:48:43.725693 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:48:43.725829 (+ 136us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:48:43.725893 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 16:48:43.726708 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:48:43.727814 (+ 1106us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:48:43.729665 (+ 1851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:48:43.730250 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:48:43.730350 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:48:43.730789 (+ 439us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 16:48:43.730896 (+ 107us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 16:48:43.730968 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0619 16:48:43.731099 (+ 131us) server_negotiation.cc:299] Negotiation successful
0619 16:48:43.731187 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260619 16:48:43.731810 23185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:48:43.721427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37670 (local address 127.0.97.62:35423)
0619 16:48:43.724980 (+ 3553us) server_negotiation.cc:206] Beginning negotiation
0619 16:48:43.724985 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:48:43.725000 (+ 15us) server_negotiation.cc:407] Connection header received
0619 16:48:43.725088 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:48:43.725091 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:48:43.725280 (+ 189us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:48:43.725449 (+ 169us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 16:48:43.726303 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:48:43.727450 (+ 1147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:48:43.729414 (+ 1964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:48:43.729961 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:48:43.730077 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:48:43.730687 (+ 610us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 16:48:43.731134 (+ 447us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 16:48:43.731214 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0619 16:48:43.731419 (+ 205us) server_negotiation.cc:299] Negotiation successful
0619 16:48:43.731501 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3453,"thread_start_us":3380,"threads_started":1}
I20260619 16:48:43.733613 23178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:48:43.725489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50646 (local address 127.0.97.60:38283)
0619 16:48:43.725639 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0619 16:48:43.725644 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:48:43.728202 (+ 2558us) server_negotiation.cc:407] Connection header received
0619 16:48:43.728352 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:48:43.728358 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:48:43.728456 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:48:43.728552 (+ 96us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 16:48:43.729120 (+ 568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:48:43.730009 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:48:43.731875 (+ 1866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:48:43.732415 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:48:43.732513 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:48:43.733001 (+ 488us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 16:48:43.733090 (+ 89us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 16:48:43.733159 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0619 16:48:43.733279 (+ 120us) server_negotiation.cc:299] Negotiation successful
0619 16:48:43.733352 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260619 16:48:43.740486 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu pid 22772
I20260619 16:48:43.741523 22834 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:48:43.848063 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu with pid 22772
W20260619 16:48:43.863973 22939 connection.cc:535] server connection from 127.0.0.1:58182 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260619 16:48:43.864295 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu pid 22846
I20260619 16:48:43.865461 22908 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:48:43.971872 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu with pid 22846
I20260619 16:48:43.987397 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu pid 22920
I20260619 16:48:43.988444 22982 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:48:44.092156 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu with pid 22920
2026-06-19T16:48:44Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (4584 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887385903558-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 16:48:44 dist-test-slave-wvz7 krb5kdc[23192](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 16:48:44 dist-test-slave-wvz7 krb5kdc[23192](info): set up 2 sockets
Jun 19 16:48:44 dist-test-slave-wvz7 krb5kdc[23192](info): commencing operation
krb5kdc: starting...
W20260619 16:48:45.094062 388 mini_kdc.cc:121] Time spent starting KDC: real 0.967s user 0.000s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887385903558-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887385903558-388-0/krb5kdc/test-user.keytab.
Jun 19 16:48:45 dist-test-slave-wvz7 krb5kdc[23192](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887725, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T16:48:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T16:48:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887385903558-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887385903558-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260619 16:48:45.312323 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu
/tmp/dist-test-taskEs2EH8/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887385903558-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887385903558-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887385903558-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887385903558-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42985
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887385903558-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44817
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:42985,127.0.97.61:36667,127.0.97.60:34355
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887385903558-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887385903558-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887385903558-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:48:45.623989 23205 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260619 16:48:45.624277 23205 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:48:45.624346 23205 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:48:45.624423 23205 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:48:45.631263 23205 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:48:45.631340 23205 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:48:45.631394 23205 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:48:45.631443 23205 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:48:45.631500 23205 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:48:45.641085 23205 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44817
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887385903558-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887385903558-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:42985,127.0.97.61:36667,127.0.97.60:34355
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887385903558-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42985
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887385903558-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887385903558-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:48:45.642356 23205 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:48:45.643633 23205 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:48:45.655377 23211 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:48:45.655709 23210 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:48:45.655334 23213 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:48:45.656656 23205 server_base.cc:1034] running on GCE node
I20260619 16:48:45.657444 23205 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:48:45.658855 23205 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:48:45.660099 23205 hybrid_clock.cc:648] HybridClock initialized: now 1781887725660029 us; error 35 us; skew 500 ppm
I20260619 16:48:45.660418 23205 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:48:45 dist-test-slave-wvz7 krb5kdc[23192](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887725, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:48:45.663568 23205 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260619 16:48:45.665023 23205 webserver.cc:460] Webserver started at http://127.0.97.62:40013/ using document root <none> and password file <none>
I20260619 16:48:45.665664 23205 fs_manager.cc:359] Metadata directory not provided
I20260619 16:48:45.665796 23205 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:48:45.666100 23205 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:48:45.669030 23205 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887385903558-388-0/minicluster-data/master-0/data/instance:
uuid: "a8932a4bca264aad92e88ef5f5a577a6"
format_stamp: "Formatted at 2026-06-19 16:48:45 on dist-test-slave-wvz7"
server_key: "bb0f9f331dac67d210f129bc297a04a9"
server_key_iv: "c5d3ad87e75a8f64f1d9ded02be07b4d"
server_key_version: "encryptionkey@0"
I20260619 16:48:45.669822 23205 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887385903558-388-0/minicluster-data/master-0/wal/instance:
uuid: "a8932a4bca264aad92e88ef5f5a577a6"
format_stamp: "Formatted at 2026-06-19 16:48:45 on dist-test-slave-wvz7"
server_key: "bb0f9f331dac67d210f129bc297a04a9"
server_key_iv: "c5d3ad87e75a8f64f1d9ded02be07b4d"
server_key_version: "encryptionkey@0"
I20260619 16:48:45.674876 23205 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260619 16:48:45.678144 23220 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:48:45.679466 23205 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260619 16:48:45.679616 23205 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887385903558-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887385903558-388-0/minicluster-data/master-0/wal
uuid: "a8932a4bca264aad92e88ef5f5a577a6"
format_stamp: "Formatted at 2026-06-19 16:48:45 on dist-test-slave-wvz7"
server_key: "bb0f9f331dac67d210f129bc297a04a9"
server_key_iv: "c5d3ad87e75a8f64f1d9ded02be07b4d"
server_key_version: "encryptionkey@0"
I20260619 16:48:45.679802 23205 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887385903558-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887385903558-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887385903558-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:48:45.701879 23205 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:48:45.705770 23205 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:48:45.706000 23205 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:48:45.722638 23205 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:42985
I20260619 16:48:45.722657 23271 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:42985 every 8 connection(s)
I20260619 16:48:45.723816 23205 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887385903558-388-0/minicluster-data/master-0/data/info.pb
I20260619 16:48:45.727327 23272 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:48:45.731473 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu as pid 23205
I20260619 16:48:45.731635 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887385903558-388-0/minicluster-data/master-0/wal/instance
I20260619 16:48:45.732009 388 external_mini_cluster.cc:1428] Setting key 9125b51937864df83adb039603502e83
I20260619 16:48:45.733500 23272 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42985 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 16:48:45 dist-test-slave-wvz7 krb5kdc[23192](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887725, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.61@KRBTEST.COM; defaulting to no policy
I20260619 16:48:45.754117 23275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:48:45.737804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46180 (local address 127.0.97.62:42985)
0619 16:48:45.738412 (+ 608us) server_negotiation.cc:206] Beginning negotiation
0619 16:48:45.738421 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 16:48:45.738475 (+ 54us) server_negotiation.cc:407] Connection header received
0619 16:48:45.739667 (+ 1192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:48:45.739672 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:48:45.739898 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:48:45.740074 (+ 176us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:48:45.741261 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:48:45.742434 (+ 1173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:48:45.743718 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:48:45.744236 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:48:45.744394 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:48:45.747715 (+ 3321us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:48:45.747723 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:48:45.747733 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:48:45.747759 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:48:45.750307 (+ 2548us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:48:45.750888 (+ 581us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:48:45.750894 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:48:45.750896 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:48:45.751028 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:48:45.751375 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:48:45.751380 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:48:45.751382 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:48:45.751811 (+ 429us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:48:45.751986 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0619 16:48:45.752492 (+ 506us) server_negotiation.cc:299] Negotiation successful
0619 16:48:45.752736 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":430,"thread_start_us":270,"threads_started":1}
I20260619 16:48:45.754133 23274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:48:45.736559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42985 (local address 127.0.0.1:46180)
0619 16:48:45.737304 (+ 745us) negotiation.cc:107] Waiting for socket to connect
0619 16:48:45.737351 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0619 16:48:45.739329 (+ 1978us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:48:45.740114 (+ 785us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:48:45.740120 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:48:45.740757 (+ 637us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:48:45.741070 (+ 313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:48:45.741086 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:48:45.742569 (+ 1483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:48:45.742573 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:48:45.743587 (+ 1014us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:48:45.743599 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:48:45.744440 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:48:45.744447 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:48:45.744667 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:48:45.745246 (+ 579us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:48:45.745268 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:48:45.747524 (+ 2256us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:48:45.750417 (+ 2893us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:48:45.750427 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:48:45.750438 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:48:45.750779 (+ 341us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:48:45.751125 (+ 346us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:48:45.751130 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:48:45.751134 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:48:45.751281 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:48:45.751934 (+ 653us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:48:45.751941 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:48:45.752148 (+ 207us) client_negotiation.cc:769] Sending connection context
0619 16:48:45.752347 (+ 199us) client_negotiation.cc:240] Negotiation successful
0619 16:48:45.752859 (+ 512us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":505,"mutex_wait_us":163,"thread_start_us":319,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.61@KRBTEST.COM" created.
I20260619 16:48:45.757018 23272 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36667 } has no permanent_uuid. Determining permanent_uuid...
I20260619 16:48:45.758049 23274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:48:45.757672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:36667 (local address 127.0.0.1:37928)
0619 16:48:45.757837 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0619 16:48:45.757933 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:36667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260619 16:48:45.758329 23222 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.61:36667: connect: Connection refused (error 111)
I20260619 16:48:45.759748 23274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:48:45.759443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:36667 (local address 127.0.0.1:37936)
0619 16:48:45.759560 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0619 16:48:45.759626 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:36667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260619 16:48:45.759929 23272 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:36667: Network error: Client connection negotiation failed: client connection to 127.0.97.61:36667: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887385903558-388-0/krb5kdc/kudu_127.0.97.61.keytab.
Entry for principal kudu/127.0.97.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887385903558-388-0/krb5kdc/kudu_127.0.97.61.keytab.
I20260619 16:48:45.780345 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu
/tmp/dist-test-taskEs2EH8/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887385903558-388-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887385903558-388-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887385903558-388-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887385903558-388-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:36667
--webserver_interface=127.0.97.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887385903558-388-0/krb5kdc/kudu_127.0.97.61.keytab
--principal=kudu/127.0.97.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44817
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:42985,127.0.97.61:36667,127.0.97.60:34355
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887385903558-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887385903558-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887385903558-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 16:48:45.809237 23272 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36667 } attempt: 1
I20260619 16:48:45.810508 23274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:48:45.810016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:36667 (local address 127.0.0.1:37942)
0619 16:48:45.810226 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0619 16:48:45.810334 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:36667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260619 16:48:45.811563 23274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:48:45.811195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:36667 (local address 127.0.0.1:37954)
0619 16:48:45.811329 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0619 16:48:45.811412 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:36667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260619 16:48:45.811779 23272 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:36667: Network error: Client connection negotiation failed: client connection to 127.0.97.61:36667: connect: Connection refused (error 111)
I20260619 16:48:45.880038 23272 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36667 } attempt: 2
I20260619 16:48:45.881363 23274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:48:45.880742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:36667 (local address 127.0.0.1:37962)
0619 16:48:45.881031 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0619 16:48:45.881156 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:36667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260619 16:48:45.882447 23274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:48:45.882051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:36667 (local address 127.0.0.1:37964)
0619 16:48:45.882193 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0619 16:48:45.882287 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:36667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260619 16:48:45.882624 23272 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:36667: Network error: Client connection negotiation failed: client connection to 127.0.97.61:36667: connect: Connection refused (error 111)
I20260619 16:48:45.973882 23272 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36667 } attempt: 3
I20260619 16:48:45.975157 23274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:48:45.974596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:36667 (local address 127.0.0.1:37968)
0619 16:48:45.974839 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0619 16:48:45.974960 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:36667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260619 16:48:45.976325 23274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:48:45.975849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:36667 (local address 127.0.0.1:37984)
0619 16:48:45.976047 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0619 16:48:45.976145 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:36667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260619 16:48:45.976522 23272 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:36667: Network error: Client connection negotiation failed: client connection to 127.0.97.61:36667: connect: Connection refused (error 111)
W20260619 16:48:46.088850 23279 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260619 16:48:46.089140 23279 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:48:46.089206 23279 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:48:46.089283 23279 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:48:46.096252 23279 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:48:46.096338 23279 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:48:46.096388 23279 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:48:46.096436 23279 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:48:46.096499 23279 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:48:46.105938 23279 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44817
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887385903558-388-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887385903558-388-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:42985,127.0.97.61:36667,127.0.97.60:34355
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887385903558-388-0/krb5kdc/kudu_127.0.97.61.keytab
--principal=kudu/127.0.97.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:36667
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887385903558-388-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.97.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887385903558-388-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:48:46.107116 23279 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:48:46.108297 23279 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:48:46.119062 23285 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:48:46.119129 23284 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:48:46.119222 23287 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:48:46.119711 23279 server_base.cc:1034] running on GCE node
I20260619 16:48:46.119732 23272 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36667 } attempt: 4
I20260619 16:48:46.120579 23279 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:48:46.120904 23274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:48:46.120322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:36667 (local address 127.0.0.1:38000)
0619 16:48:46.120617 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0619 16:48:46.120731 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:36667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":158}
I20260619 16:48:46.121850 23274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:48:46.121471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:36667 (local address 127.0.0.1:38002)
0619 16:48:46.121611 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0619 16:48:46.121695 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:36667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260619 16:48:46.122063 23279 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
W20260619 16:48:46.122087 23272 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:36667: Network error: Client connection negotiation failed: client connection to 127.0.97.61:36667: connect: Connection refused (error 111)
I20260619 16:48:46.123288 23279 hybrid_clock.cc:648] HybridClock initialized: now 1781887726123220 us; error 42 us; skew 500 ppm
I20260619 16:48:46.123651 23279 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:48:46 dist-test-slave-wvz7 krb5kdc[23192](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887726, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:48:46.126979 23279 init.cc:377] Logged in from keytab as kudu/127.0.97.61@KRBTEST.COM (short username kudu)
I20260619 16:48:46.128383 23279 webserver.cc:460] Webserver started at http://127.0.97.61:41811/ using document root <none> and password file <none>
I20260619 16:48:46.129060 23279 fs_manager.cc:359] Metadata directory not provided
I20260619 16:48:46.129166 23279 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:48:46.129475 23279 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:48:46.132385 23279 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887385903558-388-0/minicluster-data/master-1/data/instance:
uuid: "9e1408dd44dd42c6b49493e1be8ffb36"
format_stamp: "Formatted at 2026-06-19 16:48:46 on dist-test-slave-wvz7"
server_key: "d33986d6e413219bf8db0bebeffebf87"
server_key_iv: "06cf3207173d7e645f3bb9deb78a8596"
server_key_version: "encryptionkey@0"
I20260619 16:48:46.133144 23279 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887385903558-388-0/minicluster-data/master-1/wal/instance:
uuid: "9e1408dd44dd42c6b49493e1be8ffb36"
format_stamp: "Formatted at 2026-06-19 16:48:46 on dist-test-slave-wvz7"
server_key: "d33986d6e413219bf8db0bebeffebf87"
server_key_iv: "06cf3207173d7e645f3bb9deb78a8596"
server_key_version: "encryptionkey@0"
I20260619 16:48:46.138484 23279 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.006s
I20260619 16:48:46.141687 23294 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:48:46.142982 23279 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260619 16:48:46.143132 23279 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887385903558-388-0/minicluster-data/master-1/data,/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887385903558-388-0/minicluster-data/master-1/wal
uuid: "9e1408dd44dd42c6b49493e1be8ffb36"
format_stamp: "Formatted at 2026-06-19 16:48:46 on dist-test-slave-wvz7"
server_key: "d33986d6e413219bf8db0bebeffebf87"
server_key_iv: "06cf3207173d7e645f3bb9deb78a8596"
server_key_version: "encryptionkey@0"
I20260619 16:48:46.143313 23279 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887385903558-388-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887385903558-388-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887385903558-388-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:48:46.158900 23279 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:48:46.162863 23279 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:48:46.163087 23279 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:48:46.179399 23279 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.61:36667
I20260619 16:48:46.179425 23345 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.61:36667 every 8 connection(s)
I20260619 16:48:46.180661 23279 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887385903558-388-0/minicluster-data/master-1/data/info.pb
I20260619 16:48:46.182444 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu as pid 23279
I20260619 16:48:46.182611 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887385903558-388-0/minicluster-data/master-1/wal/instance
I20260619 16:48:46.183086 388 external_mini_cluster.cc:1428] Setting key f913acfcce390bb1d2f121c1c5d495ad
I20260619 16:48:46.184538 23346 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:48:46.191146 23346 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42985 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.97.60@KRBTEST.COM; defaulting to no policy
Jun 19 16:48:46 dist-test-slave-wvz7 krb5kdc[23192](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887726, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.60@KRBTEST.COM" created.
I20260619 16:48:46.211791 23275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:48:46.194183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46186 (local address 127.0.97.62:42985)
0619 16:48:46.194383 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0619 16:48:46.194390 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:48:46.195544 (+ 1154us) server_negotiation.cc:407] Connection header received
0619 16:48:46.197307 (+ 1763us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:48:46.197315 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:48:46.197399 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:48:46.197530 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:48:46.198914 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:48:46.199975 (+ 1061us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:48:46.201423 (+ 1448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:48:46.202113 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:48:46.202263 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:48:46.205694 (+ 3431us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:48:46.205699 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:48:46.205702 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:48:46.205771 (+ 69us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:48:46.208918 (+ 3147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:48:46.209585 (+ 667us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:48:46.209591 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:48:46.209594 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:48:46.209769 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:48:46.210114 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:48:46.210119 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:48:46.210122 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:48:46.210470 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:48:46.210606 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0619 16:48:46.211219 (+ 613us) server_negotiation.cc:299] Negotiation successful
0619 16:48:46.211430 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260619 16:48:46.212435 23349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:48:46.194644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42985 (local address 127.0.0.1:46186)
0619 16:48:46.195381 (+ 737us) negotiation.cc:107] Waiting for socket to connect
0619 16:48:46.195423 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0619 16:48:46.197049 (+ 1626us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:48:46.197681 (+ 632us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:48:46.197689 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:48:46.198250 (+ 561us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:48:46.198751 (+ 501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:48:46.198768 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:48:46.200115 (+ 1347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:48:46.200119 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:48:46.201263 (+ 1144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:48:46.201274 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:48:46.202321 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:48:46.202326 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:48:46.202511 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:48:46.203191 (+ 680us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:48:46.203212 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:48:46.205473 (+ 2261us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:48:46.209062 (+ 3589us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:48:46.209068 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:48:46.209079 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:48:46.209478 (+ 399us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:48:46.209868 (+ 390us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:48:46.209871 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:48:46.209873 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:48:46.210024 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:48:46.210577 (+ 553us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:48:46.210582 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:48:46.210876 (+ 294us) client_negotiation.cc:769] Sending connection context
0619 16:48:46.211088 (+ 212us) client_negotiation.cc:240] Negotiation successful
0619 16:48:46.211452 (+ 364us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":495,"thread_start_us":309,"threads_started":1}
I20260619 16:48:46.214114 23346 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36667 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 16:48:46 dist-test-slave-wvz7 krb5kdc[23192](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887726, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260619 16:48:46.228644 23349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:48:46.214873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:36667 (local address 127.0.0.1:38004)
0619 16:48:46.215074 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0619 16:48:46.215094 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 16:48:46.215210 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:48:46.215865 (+ 655us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:48:46.215876 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:48:46.216325 (+ 449us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:48:46.216471 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:48:46.216480 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:48:46.218377 (+ 1897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:48:46.218381 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:48:46.219206 (+ 825us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:48:46.219217 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:48:46.220216 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:48:46.220220 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:48:46.220351 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:48:46.220853 (+ 502us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:48:46.220870 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:48:46.222669 (+ 1799us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:48:46.226342 (+ 3673us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:48:46.226346 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:48:46.226348 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:48:46.226699 (+ 351us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:48:46.227141 (+ 442us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:48:46.227144 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:48:46.227147 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:48:46.227304 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:48:46.227971 (+ 667us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:48:46.227975 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:48:46.228107 (+ 132us) client_negotiation.cc:769] Sending connection context
0619 16:48:46.228213 (+ 106us) client_negotiation.cc:240] Negotiation successful
0619 16:48:46.228397 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":67}
I20260619 16:48:46.229038 23350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:48:46.214957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38004 (local address 127.0.97.61:36667)
0619 16:48:46.215477 (+ 520us) server_negotiation.cc:206] Beginning negotiation
0619 16:48:46.215482 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:48:46.215506 (+ 24us) server_negotiation.cc:407] Connection header received
0619 16:48:46.215545 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:48:46.215550 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:48:46.215718 (+ 168us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:48:46.215820 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:48:46.216656 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:48:46.218194 (+ 1538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:48:46.219339 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:48:46.220082 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:48:46.220192 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:48:46.222864 (+ 2672us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:48:46.222878 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:48:46.222894 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:48:46.222933 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:48:46.226211 (+ 3278us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:48:46.226829 (+ 618us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:48:46.226839 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:48:46.226843 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:48:46.226984 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:48:46.227417 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:48:46.227422 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:48:46.227425 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:48:46.227861 (+ 436us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:48:46.228073 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0619 16:48:46.228384 (+ 311us) server_negotiation.cc:299] Negotiation successful
0619 16:48:46.228639 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":391,"thread_start_us":252,"threads_started":1}
I20260619 16:48:46.231400 23346 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34355 } has no permanent_uuid. Determining permanent_uuid...
I20260619 16:48:46.232465 23349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:48:46.232064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:34355 (local address 127.0.0.1:58728)
0619 16:48:46.232210 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0619 16:48:46.232327 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:34355: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260619 16:48:46.232800 23296 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.60:34355: connect: Connection refused (error 111)
I20260619 16:48:46.234402 23349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:48:46.234023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:34355 (local address 127.0.0.1:58734)
0619 16:48:46.234181 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0619 16:48:46.234265 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:34355: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
W20260619 16:48:46.234598 23346 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:34355: Network error: Client connection negotiation failed: client connection to 127.0.97.60:34355: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887385903558-388-0/krb5kdc/kudu_127.0.97.60.keytab.
Entry for principal kudu/127.0.97.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887385903558-388-0/krb5kdc/kudu_127.0.97.60.keytab.
I20260619 16:48:46.238137 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu
/tmp/dist-test-taskEs2EH8/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887385903558-388-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887385903558-388-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887385903558-388-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887385903558-388-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:34355
--webserver_interface=127.0.97.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887385903558-388-0/krb5kdc/kudu_127.0.97.60.keytab
--principal=kudu/127.0.97.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44817
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:42985,127.0.97.61:36667,127.0.97.60:34355
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887385903558-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887385903558-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887385903558-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 16:48:46.283914 23346 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34355 } attempt: 1
I20260619 16:48:46.285305 23349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:48:46.284745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:34355 (local address 127.0.0.1:58738)
0619 16:48:46.284982 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0619 16:48:46.285102 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:34355: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260619 16:48:46.286465 23349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:48:46.286033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:34355 (local address 127.0.0.1:58750)
0619 16:48:46.286222 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0619 16:48:46.286325 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:34355: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260619 16:48:46.286681 23346 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:34355: Network error: Client connection negotiation failed: client connection to 127.0.97.60:34355: connect: Connection refused (error 111)
I20260619 16:48:46.354920 23346 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34355 } attempt: 2
I20260619 16:48:46.356096 23349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:48:46.355590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:34355 (local address 127.0.0.1:58766)
0619 16:48:46.355792 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0619 16:48:46.355912 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:34355: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260619 16:48:46.357060 23349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:48:46.356691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:34355 (local address 127.0.0.1:58772)
0619 16:48:46.356837 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0619 16:48:46.356920 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:34355: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260619 16:48:46.357259 23346 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:34355: Network error: Client connection negotiation failed: client connection to 127.0.97.60:34355: connect: Connection refused (error 111)
I20260619 16:48:46.421339 23272 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36667 } attempt: 5
Jun 19 16:48:46 dist-test-slave-wvz7 krb5kdc[23192](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887725, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260619 16:48:46.434109 23274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:48:46.422089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:36667 (local address 127.0.0.1:38014)
0619 16:48:46.422274 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0619 16:48:46.422296 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0619 16:48:46.422426 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:48:46.422805 (+ 379us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:48:46.422810 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:48:46.423196 (+ 386us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:48:46.423371 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:48:46.423381 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:48:46.424363 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:48:46.424367 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:48:46.425206 (+ 839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:48:46.425217 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:48:46.426017 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:48:46.426023 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:48:46.426203 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:48:46.426832 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:48:46.426849 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:48:46.429020 (+ 2171us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:48:46.431862 (+ 2842us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:48:46.431868 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:48:46.431872 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:48:46.432222 (+ 350us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:48:46.432559 (+ 337us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:48:46.432564 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:48:46.432568 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:48:46.432693 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:48:46.433175 (+ 482us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:48:46.433180 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:48:46.433318 (+ 138us) client_negotiation.cc:769] Sending connection context
0619 16:48:46.433433 (+ 115us) client_negotiation.cc:240] Negotiation successful
0619 16:48:46.433723 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":58}
I20260619 16:48:46.434109 23350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:48:46.422196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38014 (local address 127.0.97.61:36667)
0619 16:48:46.422397 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0619 16:48:46.422403 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:48:46.422424 (+ 21us) server_negotiation.cc:407] Connection header received
0619 16:48:46.422542 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:48:46.422548 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:48:46.422631 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:48:46.422711 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:48:46.423530 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:48:46.424214 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:48:46.425353 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:48:46.425867 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:48:46.425973 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:48:46.429213 (+ 3240us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:48:46.429219 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:48:46.429223 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:48:46.429255 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:48:46.431742 (+ 2487us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:48:46.432328 (+ 586us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:48:46.432333 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:48:46.432337 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:48:46.432453 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:48:46.432789 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:48:46.432794 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:48:46.432798 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:48:46.433071 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:48:46.433196 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0619 16:48:46.433540 (+ 344us) server_negotiation.cc:299] Negotiation successful
0619 16:48:46.433795 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260619 16:48:46.435139 23272 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34355 } has no permanent_uuid. Determining permanent_uuid...
I20260619 16:48:46.436005 23274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:48:46.435663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:34355 (local address 127.0.0.1:58780)
0619 16:48:46.435807 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0619 16:48:46.435877 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:34355: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260619 16:48:46.436805 23274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:48:46.436495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:34355 (local address 127.0.0.1:58784)
0619 16:48:46.436629 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0619 16:48:46.436716 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:34355: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260619 16:48:46.436970 23272 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:34355: Network error: Client connection negotiation failed: client connection to 127.0.97.60:34355: connect: Connection refused (error 111)
I20260619 16:48:46.448450 23346 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34355 } attempt: 3
I20260619 16:48:46.449366 23349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:48:46.448936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:34355 (local address 127.0.0.1:58792)
0619 16:48:46.449119 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0619 16:48:46.449209 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:34355: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260619 16:48:46.450307 23349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:48:46.449894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:34355 (local address 127.0.0.1:58804)
0619 16:48:46.450099 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0619 16:48:46.450180 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:34355: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110}
W20260619 16:48:46.450520 23346 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:34355: Network error: Client connection negotiation failed: client connection to 127.0.97.60:34355: connect: Connection refused (error 111)
I20260619 16:48:46.488193 23272 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34355 } attempt: 1
I20260619 16:48:46.489315 23274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:48:46.488857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:34355 (local address 127.0.0.1:58820)
0619 16:48:46.489055 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0619 16:48:46.489149 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:34355: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260619 16:48:46.490525 23274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:48:46.490203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:34355 (local address 127.0.0.1:58826)
0619 16:48:46.490340 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0619 16:48:46.490412 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:34355: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260619 16:48:46.490741 23272 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:34355: Network error: Client connection negotiation failed: client connection to 127.0.97.60:34355: connect: Connection refused (error 111)
W20260619 16:48:46.547030 23353 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260619 16:48:46.547369 23353 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:48:46.547469 23353 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:48:46.547581 23353 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:48:46.554579 23353 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:48:46.554668 23353 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:48:46.554744 23353 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:48:46.554811 23353 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:48:46.554893 23353 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:48:46.558934 23272 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34355 } attempt: 2
I20260619 16:48:46.560042 23274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:48:46.559579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:34355 (local address 127.0.0.1:58838)
0619 16:48:46.559805 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0619 16:48:46.559891 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:34355: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20260619 16:48:46.561028 23274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:48:46.560706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:34355 (local address 127.0.0.1:58840)
0619 16:48:46.560848 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0619 16:48:46.560917 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:34355: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260619 16:48:46.561218 23272 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:34355: Network error: Client connection negotiation failed: client connection to 127.0.97.60:34355: connect: Connection refused (error 111)
I20260619 16:48:46.564894 23353 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44817
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887385903558-388-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887385903558-388-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:42985,127.0.97.61:36667,127.0.97.60:34355
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887385903558-388-0/krb5kdc/kudu_127.0.97.60.keytab
--principal=kudu/127.0.97.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:34355
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887385903558-388-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.97.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887385903558-388-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:48:46.566145 23353 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:48:46.567351 23353 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:48:46.577948 23358 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:48:46.577927 23359 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:48:46.578456 23361 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:48:46.578686 23353 server_base.cc:1034] running on GCE node
I20260619 16:48:46.579392 23353 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:48:46.580749 23353 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:48:46.581969 23353 hybrid_clock.cc:648] HybridClock initialized: now 1781887726581903 us; error 43 us; skew 500 ppm
I20260619 16:48:46.582288 23353 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:48:46 dist-test-slave-wvz7 krb5kdc[23192](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887726, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:48:46.585354 23353 init.cc:377] Logged in from keytab as kudu/127.0.97.60@KRBTEST.COM (short username kudu)
I20260619 16:48:46.586702 23353 webserver.cc:460] Webserver started at http://127.0.97.60:38385/ using document root <none> and password file <none>
I20260619 16:48:46.587332 23353 fs_manager.cc:359] Metadata directory not provided
I20260619 16:48:46.587411 23353 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:48:46.587658 23353 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:48:46.590471 23353 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887385903558-388-0/minicluster-data/master-2/data/instance:
uuid: "bc1be13db59442abac3d356cab9f7668"
format_stamp: "Formatted at 2026-06-19 16:48:46 on dist-test-slave-wvz7"
server_key: "350fbb7edee16333a94a17fc69d0a1d4"
server_key_iv: "0e0f06d0ef503c685fa9c5ccbef59c8b"
server_key_version: "encryptionkey@0"
I20260619 16:48:46.591171 23353 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887385903558-388-0/minicluster-data/master-2/wal/instance:
uuid: "bc1be13db59442abac3d356cab9f7668"
format_stamp: "Formatted at 2026-06-19 16:48:46 on dist-test-slave-wvz7"
server_key: "350fbb7edee16333a94a17fc69d0a1d4"
server_key_iv: "0e0f06d0ef503c685fa9c5ccbef59c8b"
server_key_version: "encryptionkey@0"
I20260619 16:48:46.593768 23346 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34355 } attempt: 4
I20260619 16:48:46.594916 23349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:48:46.594355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:34355 (local address 127.0.0.1:58844)
0619 16:48:46.594584 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0619 16:48:46.594691 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:34355: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260619 16:48:46.595885 23349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:48:46.595548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:34355 (local address 127.0.0.1:58858)
0619 16:48:46.595708 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0619 16:48:46.595787 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:34355: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260619 16:48:46.596115 23346 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:34355: Network error: Client connection negotiation failed: client connection to 127.0.97.60:34355: connect: Connection refused (error 111)
I20260619 16:48:46.596382 23353 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 16:48:46.599606 23368 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:48:46.600847 23353 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 16:48:46.600989 23353 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887385903558-388-0/minicluster-data/master-2/data,/tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887385903558-388-0/minicluster-data/master-2/wal
uuid: "bc1be13db59442abac3d356cab9f7668"
format_stamp: "Formatted at 2026-06-19 16:48:46 on dist-test-slave-wvz7"
server_key: "350fbb7edee16333a94a17fc69d0a1d4"
server_key_iv: "0e0f06d0ef503c685fa9c5ccbef59c8b"
server_key_version: "encryptionkey@0"
I20260619 16:48:46.601181 23353 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887385903558-388-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887385903558-388-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887385903558-388-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:48:46.613626 23353 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:48:46.617532 23353 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:48:46.617790 23353 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:48:46.633623 23353 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.60:34355
I20260619 16:48:46.633642 23419 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.60:34355 every 8 connection(s)
I20260619 16:48:46.634909 23353 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887385903558-388-0/minicluster-data/master-2/data/info.pb
I20260619 16:48:46.638430 23420 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:48:46.639956 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu as pid 23353
I20260619 16:48:46.640105 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEs2EH8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887385903558-388-0/minicluster-data/master-2/wal/instance
I20260619 16:48:46.640494 388 external_mini_cluster.cc:1428] Setting key 1f259154f4cb491983603dd643fa8bfe
I20260619 16:48:46.647173 23420 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42985 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 16:48:46 dist-test-slave-wvz7 krb5kdc[23192](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887725, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
Jun 19 16:48:46 dist-test-slave-wvz7 krb5kdc[23192](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887726, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 16:48:46.661329 23423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:48:46.642077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58874 (local address 127.0.97.60:34355)
0619 16:48:46.642809 (+ 732us) server_negotiation.cc:206] Beginning negotiation
0619 16:48:46.642823 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 16:48:46.642860 (+ 37us) server_negotiation.cc:407] Connection header received
0619 16:48:46.644354 (+ 1494us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:48:46.644374 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:48:46.644749 (+ 375us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:48:46.645048 (+ 299us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:48:46.646691 (+ 1643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:48:46.648064 (+ 1373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:48:46.649629 (+ 1565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:48:46.650667 (+ 1038us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:48:46.651167 (+ 500us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:48:46.654719 (+ 3552us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:48:46.654726 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:48:46.654737 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:48:46.654778 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:48:46.657566 (+ 2788us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:48:46.658235 (+ 669us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:48:46.658241 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:48:46.658247 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:48:46.658389 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:48:46.658745 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:48:46.658749 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:48:46.658751 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:48:46.659212 (+ 461us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:48:46.659481 (+ 269us) server_negotiation.cc:1035] Waiting for connection context
0619 16:48:46.659893 (+ 412us) server_negotiation.cc:299] Negotiation successful
0619 16:48:46.660252 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":535,"thread_start_us":332,"threads_started":1}
I20260619 16:48:46.662643 23424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:48:46.649606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42985 (local address 127.0.0.1:46200)
0619 16:48:46.650307 (+ 701us) negotiation.cc:107] Waiting for socket to connect
0619 16:48:46.650343 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0619 16:48:46.650472 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:48:46.650800 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:48:46.650806 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:48:46.651367 (+ 561us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:48:46.651663 (+ 296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:48:46.651673 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:48:46.652579 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:48:46.652585 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:48:46.653884 (+ 1299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:48:46.653898 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:48:46.654666 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:48:46.654674 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:48:46.654825 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:48:46.655394 (+ 569us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:48:46.655417 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:48:46.657675 (+ 2258us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:48:46.660347 (+ 2672us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:48:46.660358 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:48:46.660365 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:48:46.660748 (+ 383us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:48:46.661127 (+ 379us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:48:46.661132 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:48:46.661136 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:48:46.661256 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:48:46.661820 (+ 564us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:48:46.661825 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:48:46.661977 (+ 152us) client_negotiation.cc:769] Sending connection context
0619 16:48:46.662111 (+ 134us) client_negotiation.cc:240] Negotiation successful
0619 16:48:46.662346 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":513,"thread_start_us":383,"threads_started":1}
I20260619 16:48:46.662791 23275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:48:46.649727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46200 (local address 127.0.97.62:42985)
0619 16:48:46.649931 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0619 16:48:46.649937 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:48:46.650425 (+ 488us) server_negotiation.cc:407] Connection header received
0619 16:48:46.650585 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:48:46.650590 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:48:46.650654 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:48:46.650730 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:48:46.651794 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:48:46.652447 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:48:46.654014 (+ 1567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:48:46.654435 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:48:46.654544 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:48:46.657874 (+ 3330us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:48:46.657878 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:48:46.657881 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:48:46.657911 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:48:46.660240 (+ 2329us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:48:46.660910 (+ 670us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:48:46.660914 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:48:46.660916 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:48:46.661028 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:48:46.661359 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:48:46.661364 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:48:46.661368 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:48:46.661675 (+ 307us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:48:46.661855 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0619 16:48:46.662256 (+ 401us) server_negotiation.cc:299] Negotiation successful
0619 16:48:46.662454 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260619 16:48:46.664587 23420 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36667 } has no permanent_uuid. Determining permanent_uuid...
I20260619 16:48:46.667435 23272 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34355 } attempt: 3
Jun 19 16:48:46 dist-test-slave-wvz7 krb5kdc[23192](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887725, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
Jun 19 16:48:46 dist-test-slave-wvz7 krb5kdc[23192](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887726, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
Jun 19 16:48:46 dist-test-slave-wvz7 krb5kdc[23192](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887725, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
I20260619 16:48:46.679409 23424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:48:46.665329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:36667 (local address 127.0.0.1:38016)
0619 16:48:46.665468 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0619 16:48:46.665481 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 16:48:46.665548 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:48:46.666255 (+ 707us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:48:46.666258 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:48:46.666648 (+ 390us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:48:46.666813 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:48:46.666824 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:48:46.668113 (+ 1289us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:48:46.668118 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:48:46.669477 (+ 1359us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:48:46.669491 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:48:46.670578 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:48:46.670583 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:48:46.670766 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:48:46.671277 (+ 511us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:48:46.671297 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:48:46.673481 (+ 2184us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:48:46.676627 (+ 3146us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:48:46.676631 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:48:46.676633 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:48:46.677039 (+ 406us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:48:46.677649 (+ 610us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:48:46.677652 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:48:46.677655 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:48:46.677855 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:48:46.678460 (+ 605us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:48:46.678464 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:48:46.678611 (+ 147us) client_negotiation.cc:769] Sending connection context
0619 16:48:46.678780 (+ 169us) client_negotiation.cc:240] Negotiation successful
0619 16:48:46.679050 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51,"mutex_wait_us":3}
I20260619 16:48:46.679452 23350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:48:46.665788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38016 (local address 127.0.97.61:36667)
0619 16:48:46.665968 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0619 16:48:46.665974 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:48:46.665993 (+ 19us) server_negotiation.cc:407] Connection header received
0619 16:48:46.666050 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:48:46.666055 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:48:46.666139 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:48:46.666236 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:48:46.666949 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:48:46.667767 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:48:46.669629 (+ 1862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:48:46.670275 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:48:46.670361 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:48:46.673674 (+ 3313us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:48:46.673694 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:48:46.673699 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:48:46.673762 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:48:46.676476 (+ 2714us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:48:46.677163 (+ 687us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:48:46.677169 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:48:46.677172 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:48:46.677332 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:48:46.677976 (+ 644us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:48:46.677982 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:48:46.677985 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:48:46.678317 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:48:46.678466 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0619 16:48:46.678882 (+ 416us) server_negotiation.cc:299] Negotiation successful
0619 16:48:46.679097 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260619 16:48:46.680166 23275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:48:46.664276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46212 (local address 127.0.97.62:42985)
0619 16:48:46.664434 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0619 16:48:46.664440 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:48:46.664665 (+ 225us) server_negotiation.cc:407] Connection header received
0619 16:48:46.664820 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:48:46.664825 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:48:46.664896 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:48:46.665010 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:48:46.666380 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:48:46.667302 (+ 922us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:48:46.668980 (+ 1678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:48:46.669437 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:48:46.669542 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:48:46.673562 (+ 4020us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:48:46.673569 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:48:46.673573 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:48:46.673618 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:48:46.676901 (+ 3283us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:48:46.677533 (+ 632us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:48:46.677539 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:48:46.677542 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:48:46.677685 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:48:46.678092 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:48:46.678096 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:48:46.678098 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:48:46.678359 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:48:46.678607 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0619 16:48:46.679678 (+ 1071us) server_negotiation.cc:299] Negotiation successful
0619 16:48:46.679867 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260619 16:48:46.680922 23420 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34355 } has no permanent_uuid. Determining permanent_uuid...
I20260619 16:48:46.681500 23274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:48:46.667879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:34355 (local address 127.0.0.1:58876)
0619 16:48:46.668038 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0619 16:48:46.668057 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 16:48:46.668178 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:48:46.668446 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:48:46.668450 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:48:46.668827 (+ 377us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:48:46.668968 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:48:46.668978 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:48:46.670245 (+ 1267us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:48:46.670251 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:48:46.671492 (+ 1241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:48:46.671507 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:48:46.672498 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:48:46.672504 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:48:46.672702 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:48:46.673228 (+ 526us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:48:46.673251 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:48:46.675605 (+ 2354us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:48:46.679064 (+ 3459us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:48:46.679069 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:48:46.679072 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:48:46.679499 (+ 427us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:48:46.679870 (+ 371us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:48:46.679875 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:48:46.679879 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:48:46.680029 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:48:46.680579 (+ 550us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:48:46.680584 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:48:46.680800 (+ 216us) client_negotiation.cc:769] Sending connection context
0619 16:48:46.680927 (+ 127us) client_negotiation.cc:240] Negotiation successful
0619 16:48:46.681182 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46}
I20260619 16:48:46.682240 23423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:48:46.667982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58876 (local address 127.0.97.60:34355)
0619 16:48:46.668119 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0619 16:48:46.668123 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:48:46.668140 (+ 17us) server_negotiation.cc:407] Connection header received
0619 16:48:46.668279 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:48:46.668284 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:48:46.668340 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:48:46.668410 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:48:46.669102 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:48:46.670102 (+ 1000us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:48:46.671684 (+ 1582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:48:46.672321 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:48:46.672446 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:48:46.675818 (+ 3372us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:48:46.675824 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:48:46.675829 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:48:46.675863 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:48:46.678920 (+ 3057us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:48:46.679651 (+ 731us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:48:46.679655 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:48:46.679658 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:48:46.679769 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:48:46.680117 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:48:46.680122 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:48:46.680125 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:48:46.680445 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:48:46.681716 (+ 1271us) server_negotiation.cc:1035] Waiting for connection context
0619 16:48:46.681861 (+ 145us) server_negotiation.cc:299] Negotiation successful
0619 16:48:46.682024 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260619 16:48:46.688059 23272 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a8932a4bca264aad92e88ef5f5a577a6: Bootstrap starting.
Jun 19 16:48:46 dist-test-slave-wvz7 krb5kdc[23192](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887726, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
I20260619 16:48:46.691973 23272 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a8932a4bca264aad92e88ef5f5a577a6: Neither blocks nor log segments found. Creating new log.
Jun 19 16:48:46 dist-test-slave-wvz7 krb5kdc[23192](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887725, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260619 16:48:46.693113 23272 log.cc:824] T 00000000000000000000000000000000 P a8932a4bca264aad92e88ef5f5a577a6: Log is configured to *not* fsync() on all Append() calls
I20260619 16:48:46.694190 23424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:48:46.682079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:34355 (local address 127.0.0.1:58892)
0619 16:48:46.682215 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0619 16:48:46.682235 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 16:48:46.682326 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:48:46.682688 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:48:46.682693 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:48:46.683130 (+ 437us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:48:46.683300 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:48:46.683308 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:48:46.684571 (+ 1263us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:48:46.684576 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:48:46.685615 (+ 1039us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:48:46.685626 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:48:46.686282 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:48:46.686288 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:48:46.686478 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:48:46.686928 (+ 450us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:48:46.686943 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:48:46.688889 (+ 1946us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:48:46.691531 (+ 2642us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:48:46.691536 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:48:46.691540 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:48:46.691988 (+ 448us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:48:46.692378 (+ 390us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:48:46.692383 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:48:46.692387 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:48:46.692537 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:48:46.693163 (+ 626us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:48:46.693167 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:48:46.693307 (+ 140us) client_negotiation.cc:769] Sending connection context
0619 16:48:46.693462 (+ 155us) client_negotiation.cc:240] Negotiation successful
0619 16:48:46.693800 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45}
I20260619 16:48:46.696095 23423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:48:46.682255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58892 (local address 127.0.97.60:34355)
0619 16:48:46.682419 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0619 16:48:46.682424 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:48:46.682439 (+ 15us) server_negotiation.cc:407] Connection header received
0619 16:48:46.682504 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:48:46.682508 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:48:46.682568 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:48:46.682648 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:48:46.683423 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:48:46.684435 (+ 1012us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:48:46.685748 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:48:46.686165 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:48:46.686262 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:48:46.689223 (+ 2961us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:48:46.689227 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:48:46.689230 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:48:46.689255 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:48:46.691424 (+ 2169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:48:46.692128 (+ 704us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:48:46.692132 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:48:46.692136 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:48:46.692263 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:48:46.692655 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:48:46.692660 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:48:46.692663 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:48:46.693030 (+ 367us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:48:46.693169 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0619 16:48:46.693589 (+ 420us) server_negotiation.cc:299] Negotiation successful
0619 16:48:46.694625 (+ 1036us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":135,"server-negotiator.queue_time_us":77,"thread_start_us":340,"threads_started":1}
I20260619 16:48:46.698285 23350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:48:46.681646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38028 (local address 127.0.97.61:36667)
0619 16:48:46.681813 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0619 16:48:46.681818 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:48:46.683207 (+ 1389us) server_negotiation.cc:407] Connection header received
0619 16:48:46.683413 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:48:46.683418 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:48:46.683488 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:48:46.683573 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:48:46.685202 (+ 1629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:48:46.686238 (+ 1036us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:48:46.687808 (+ 1570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:48:46.688416 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:48:46.688522 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:48:46.693161 (+ 4639us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:48:46.693167 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:48:46.693172 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:48:46.693207 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:48:46.695687 (+ 2480us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:48:46.696406 (+ 719us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:48:46.696412 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:48:46.696416 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:48:46.696595 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:48:46.696960 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:48:46.696965 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:48:46.696969 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:48:46.697234 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:48:46.697473 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0619 16:48:46.697773 (+ 300us) server_negotiation.cc:299] Negotiation successful
0619 16:48:46.697979 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260619 16:48:46.701728 23420 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bc1be13db59442abac3d356cab9f7668: Bootstrap starting.
I20260619 16:48:46.702183 23272 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a8932a4bca264aad92e88ef5f5a577a6: No bootstrap required, opened a new log
I20260619 16:48:46.702677 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 16:48:46.705367 23420 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bc1be13db59442abac3d356cab9f7668: Neither blocks nor log segments found. Creating new log.
I20260619 16:48:46.705499 23247 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "a8932a4bca264aad92e88ef5f5a577a6"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46212
I20260619 16:48:46.706461 23420 log.cc:824] T 00000000000000000000000000000000 P bc1be13db59442abac3d356cab9f7668: Log is configured to *not* fsync() on all Append() calls
I20260619 16:48:46.708830 23420 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bc1be13db59442abac3d356cab9f7668: No bootstrap required, opened a new log
I20260619 16:48:46.711596 23272 raft_consensus.cc:348] T 00000000000000000000000000000000 P a8932a4bca264aad92e88ef5f5a577a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8932a4bca264aad92e88ef5f5a577a6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42985 } } peers { permanent_uuid: "9e1408dd44dd42c6b49493e1be8ffb36" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36667 } } peers { permanent_uuid: "bc1be13db59442abac3d356cab9f7668" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34355 } }
I20260619 16:48:46.711934 23272 raft_consensus.cc:729] T 00000000000000000000000000000000 P a8932a4bca264aad92e88ef5f5a577a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a8932a4bca264aad92e88ef5f5a577a6, State: Initialized, Role: FOLLOWER
I20260619 16:48:46.712349 23272 consensus_queue.cc:260] T 00000000000000000000000000000000 P a8932a4bca264aad92e88ef5f5a577a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8932a4bca264aad92e88ef5f5a577a6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42985 } } peers { permanent_uuid: "9e1408dd44dd42c6b49493e1be8ffb36" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36667 } } peers { permanent_uuid: "bc1be13db59442abac3d356cab9f7668" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34355 } }
I20260619 16:48:46.713162 23428 sys_catalog.cc:455] T 00000000000000000000000000000000 P a8932a4bca264aad92e88ef5f5a577a6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8932a4bca264aad92e88ef5f5a577a6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42985 } } peers { permanent_uuid: "9e1408dd44dd42c6b49493e1be8ffb36" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36667 } } peers { permanent_uuid: "bc1be13db59442abac3d356cab9f7668" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34355 } } }
I20260619 16:48:46.713627 23428 sys_catalog.cc:458] T 00000000000000000000000000000000 P a8932a4bca264aad92e88ef5f5a577a6 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 16:48:46.714371 23272 sys_catalog.cc:564] T 00000000000000000000000000000000 P a8932a4bca264aad92e88ef5f5a577a6 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:48:46.714442 23247 raft_consensus.cc:482] T 00000000000000000000000000000000 P a8932a4bca264aad92e88ef5f5a577a6 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260619 16:48:46.714623 23247 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a8932a4bca264aad92e88ef5f5a577a6 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:48:46.716364 23420 raft_consensus.cc:348] T 00000000000000000000000000000000 P bc1be13db59442abac3d356cab9f7668 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8932a4bca264aad92e88ef5f5a577a6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42985 } } peers { permanent_uuid: "9e1408dd44dd42c6b49493e1be8ffb36" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36667 } } peers { permanent_uuid: "bc1be13db59442abac3d356cab9f7668" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34355 } }
I20260619 16:48:46.716625 23420 raft_consensus.cc:729] T 00000000000000000000000000000000 P bc1be13db59442abac3d356cab9f7668 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc1be13db59442abac3d356cab9f7668, State: Initialized, Role: FOLLOWER
I20260619 16:48:46.717006 23420 consensus_queue.cc:260] T 00000000000000000000000000000000 P bc1be13db59442abac3d356cab9f7668 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8932a4bca264aad92e88ef5f5a577a6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42985 } } peers { permanent_uuid: "9e1408dd44dd42c6b49493e1be8ffb36" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36667 } } peers { permanent_uuid: "bc1be13db59442abac3d356cab9f7668" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34355 } }
I20260619 16:48:46.717806 23431 sys_catalog.cc:455] T 00000000000000000000000000000000 P bc1be13db59442abac3d356cab9f7668 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8932a4bca264aad92e88ef5f5a577a6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42985 } } peers { permanent_uuid: "9e1408dd44dd42c6b49493e1be8ffb36" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36667 } } peers { permanent_uuid: "bc1be13db59442abac3d356cab9f7668" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34355 } } }
I20260619 16:48:46.718200 23431 sys_catalog.cc:458] T 00000000000000000000000000000000 P bc1be13db59442abac3d356cab9f7668 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 16:48:46.718029 23247 raft_consensus.cc:504] T 00000000000000000000000000000000 P a8932a4bca264aad92e88ef5f5a577a6 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8932a4bca264aad92e88ef5f5a577a6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42985 } } peers { permanent_uuid: "9e1408dd44dd42c6b49493e1be8ffb36" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36667 } } peers { permanent_uuid: "bc1be13db59442abac3d356cab9f7668" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34355 } }
I20260619 16:48:46.718815 23420 sys_catalog.cc:564] T 00000000000000000000000000000000 P bc1be13db59442abac3d356cab9f7668 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:48:46.719458 23247 leader_election.cc:288] T 00000000000000000000000000000000 P a8932a4bca264aad92e88ef5f5a577a6 [CANDIDATE]: Term 1 election: Requested vote from peers 9e1408dd44dd42c6b49493e1be8ffb36 (127.0.97.61:36667), bc1be13db59442abac3d356cab9f7668 (127.0.97.60:34355)
I20260619 16:48:46.720341 23395 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a8932a4bca264aad92e88ef5f5a577a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "bc1be13db59442abac3d356cab9f7668"
I20260619 16:48:46.720818 23395 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bc1be13db59442abac3d356cab9f7668 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:48:46.724103 23395 raft_consensus.cc:2454] T 00000000000000000000000000000000 P bc1be13db59442abac3d356cab9f7668 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a8932a4bca264aad92e88ef5f5a577a6 in term 1.
I20260619 16:48:46.725024 23222 leader_election.cc:302] T 00000000000000000000000000000000 P a8932a4bca264aad92e88ef5f5a577a6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a8932a4bca264aad92e88ef5f5a577a6, bc1be13db59442abac3d356cab9f7668; no voters:
I20260619 16:48:46.725442 23428 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a8932a4bca264aad92e88ef5f5a577a6 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:48:46.745467 23428 raft_consensus.cc:686] T 00000000000000000000000000000000 P a8932a4bca264aad92e88ef5f5a577a6 [term 1 LEADER]: Becoming Leader. State: Replica: a8932a4bca264aad92e88ef5f5a577a6, State: Running, Role: LEADER
I20260619 16:48:46.746572 23428 consensus_queue.cc:237] T 00000000000000000000000000000000 P a8932a4bca264aad92e88ef5f5a577a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8932a4bca264aad92e88ef5f5a577a6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42985 } } peers { permanent_uuid: "9e1408dd44dd42c6b49493e1be8ffb36" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36667 } } peers { permanent_uuid: "bc1be13db59442abac3d356cab9f7668" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34355 } }
I20260619 16:48:46.760301 23437 sys_catalog.cc:455] T 00000000000000000000000000000000 P a8932a4bca264aad92e88ef5f5a577a6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a8932a4bca264aad92e88ef5f5a577a6. Latest consensus state: current_term: 1 leader_uuid: "a8932a4bca264aad92e88ef5f5a577a6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8932a4bca264aad92e88ef5f5a577a6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42985 } } peers { permanent_uuid: "9e1408dd44dd42c6b49493e1be8ffb36" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36667 } } peers { permanent_uuid: "bc1be13db59442abac3d356cab9f7668" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34355 } } }
I20260619 16:48:46.760771 23437 sys_catalog.cc:458] T 00000000000000000000000000000000 P a8932a4bca264aad92e88ef5f5a577a6 [sys.catalog]: This master's current role is: LEADER
I20260619 16:48:46.774446 23455 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:48:46.792976 23455 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:48:46.815433 23350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:48:46.742676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38040 (local address 127.0.97.61:36667)
0619 16:48:46.743013 (+ 337us) server_negotiation.cc:206] Beginning negotiation
0619 16:48:46.743019 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:48:46.772101 (+ 29082us) server_negotiation.cc:407] Connection header received
0619 16:48:46.773372 (+ 1271us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:48:46.773379 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:48:46.773467 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:48:46.773540 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:48:46.775513 (+ 1973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:48:46.776529 (+ 1016us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:48:46.781508 (+ 4979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:48:46.782199 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:48:46.782297 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:48:46.794053 (+ 11756us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:48:46.794061 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:48:46.794066 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:48:46.794108 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:48:46.797471 (+ 3363us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:48:46.811797 (+ 14326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:48:46.811804 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:48:46.811808 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:48:46.811965 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:48:46.813464 (+ 1499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:48:46.813470 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:48:46.813473 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:48:46.813896 (+ 423us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:48:46.814168 (+ 272us) server_negotiation.cc:1035] Waiting for connection context
0619 16:48:46.814754 (+ 586us) server_negotiation.cc:299] Negotiation successful
0619 16:48:46.815020 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":210}
I20260619 16:48:46.819407 23275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:48:46.742506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46222 (local address 127.0.97.62:42985)
0619 16:48:46.742861 (+ 355us) server_negotiation.cc:206] Beginning negotiation
0619 16:48:46.742867 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:48:46.764918 (+ 22051us) server_negotiation.cc:407] Connection header received
0619 16:48:46.766043 (+ 1125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:48:46.766049 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:48:46.766147 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:48:46.766223 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:48:46.768381 (+ 2158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:48:46.769477 (+ 1096us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:48:46.782511 (+ 13034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:48:46.783261 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:48:46.783360 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:48:46.794941 (+ 11581us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:48:46.794953 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:48:46.794959 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:48:46.795004 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:48:46.800424 (+ 5420us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:48:46.811082 (+ 10658us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:48:46.811088 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:48:46.811092 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:48:46.811289 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:48:46.812386 (+ 1097us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:48:46.812392 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:48:46.812396 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:48:46.812764 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:48:46.813028 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0619 16:48:46.818754 (+ 5726us) server_negotiation.cc:299] Negotiation successful
0619 16:48:46.819020 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":231}
I20260619 16:48:46.800130 23353 master_runner.cc:186] Error getting master registration for 127.0.97.62:42985: OK, instance_id { permanent_uuid: "a8932a4bca264aad92e88ef5f5a577a6" instance_seqno: 1781887725717971 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260619 16:48:46.826380 23423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:48:46.765564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58896 (local address 127.0.97.60:34355)
0619 16:48:46.765774 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0619 16:48:46.765780 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:48:46.778845 (+ 13065us) server_negotiation.cc:407] Connection header received
0619 16:48:46.779079 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:48:46.779096 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:48:46.779203 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:48:46.779273 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:48:46.785072 (+ 5799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:48:46.786145 (+ 1073us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:48:46.791961 (+ 5816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:48:46.792667 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:48:46.792766 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:48:46.796809 (+ 4043us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:48:46.796816 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:48:46.796820 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:48:46.796865 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:48:46.805682 (+ 8817us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:48:46.824014 (+ 18332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:48:46.824020 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:48:46.824024 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:48:46.824180 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:48:46.824594 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:48:46.824600 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:48:46.824603 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:48:46.825000 (+ 397us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:48:46.825239 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0619 16:48:46.825775 (+ 536us) server_negotiation.cc:299] Negotiation successful
0619 16:48:46.826042 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260619 16:48:46.829115 23353 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260619 16:48:46.837869 23458 catalog_manager.cc:1558] T 00000000000000000000000000000000 P bc1be13db59442abac3d356cab9f7668: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 16:48:46.837984 23458 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 16:48:46.838179 23395 raft_consensus.cc:1261] T 00000000000000000000000000000000 P bc1be13db59442abac3d356cab9f7668 [term 1 FOLLOWER]: Refusing update from remote peer a8932a4bca264aad92e88ef5f5a577a6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:48:46.839318 23428 consensus_queue.cc:1035] T 00000000000000000000000000000000 P a8932a4bca264aad92e88ef5f5a577a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bc1be13db59442abac3d356cab9f7668" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:48:46.858810 23431 sys_catalog.cc:455] T 00000000000000000000000000000000 P bc1be13db59442abac3d356cab9f7668 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a8932a4bca264aad92e88ef5f5a577a6. Latest consensus state: current_term: 1 leader_uuid: "a8932a4bca264aad92e88ef5f5a577a6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8932a4bca264aad92e88ef5f5a577a6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42985 } } peers { permanent_uuid: "9e1408dd44dd42c6b49493e1be8ffb36" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36667 } } peers { permanent_uuid: "bc1be13db59442abac3d356cab9f7668" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34355 } } }
I20260619 16:48:46.859187 23431 sys_catalog.cc:458] T 00000000000000000000000000000000 P bc1be13db59442abac3d356cab9f7668 [sys.catalog]: This master's current role is: FOLLOWER
W20260619 16:48:46.869716 23374 tablet.cc:2369] T 00000000000000000000000000000000 P bc1be13db59442abac3d356cab9f7668: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 16:48:46.872363 23464 mvcc.cc:205] Tried to move back new op lower bound from 7298612129114783744 to 7298612128764530688. Current Snapshot: MvccSnapshot[applied={T|T < 7298612129114783744}]
I20260619 16:48:46.879936 23431 sys_catalog.cc:455] T 00000000000000000000000000000000 P bc1be13db59442abac3d356cab9f7668 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a8932a4bca264aad92e88ef5f5a577a6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8932a4bca264aad92e88ef5f5a577a6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42985 } } peers { permanent_uuid: "9e1408dd44dd42c6b49493e1be8ffb36" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36667 } } peers { permanent_uuid: "bc1be13db59442abac3d356cab9f7668" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34355 } } }
I20260619 16:48:46.880219 23431 sys_catalog.cc:458] T 00000000000000000000000000000000 P bc1be13db59442abac3d356cab9f7668 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 16:48:46.888029 23455 catalog_manager.cc:1347] Generated new cluster ID: 4dd204c7646d460fbb4beda09a62095f
I20260619 16:48:46.888145 23455 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:48:46.889288 23428 sys_catalog.cc:455] T 00000000000000000000000000000000 P a8932a4bca264aad92e88ef5f5a577a6 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a8932a4bca264aad92e88ef5f5a577a6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8932a4bca264aad92e88ef5f5a577a6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42985 } } peers { permanent_uuid: "9e1408dd44dd42c6b49493e1be8ffb36" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36667 } } peers { permanent_uuid: "bc1be13db59442abac3d356cab9f7668" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34355 } } }
I20260619 16:48:46.889602 23428 sys_catalog.cc:458] T 00000000000000000000000000000000 P a8932a4bca264aad92e88ef5f5a577a6 [sys.catalog]: This master's current role is: LEADER
I20260619 16:48:46.895314 23346 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34355 } attempt: 5
I20260619 16:48:46.895190 23468 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
Jun 19 16:48:46 dist-test-slave-wvz7 krb5kdc[23192](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887726, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
I20260619 16:48:46.925881 23349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:48:46.896298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:34355 (local address 127.0.0.1:58898)
0619 16:48:46.896619 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0619 16:48:46.896639 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 16:48:46.896741 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:48:46.897512 (+ 771us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:48:46.897517 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:48:46.898055 (+ 538us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:48:46.898257 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:48:46.898269 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:48:46.899970 (+ 1701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:48:46.899976 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:48:46.901373 (+ 1397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:48:46.901389 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:48:46.905990 (+ 4601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:48:46.905997 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:48:46.906216 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:48:46.907171 (+ 955us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:48:46.907194 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:48:46.918564 (+ 11370us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:48:46.922876 (+ 4312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:48:46.922882 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:48:46.922886 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:48:46.923464 (+ 578us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:48:46.923927 (+ 463us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:48:46.923932 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:48:46.923936 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:48:46.924099 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:48:46.924823 (+ 724us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:48:46.924828 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:48:46.924995 (+ 167us) client_negotiation.cc:769] Sending connection context
0619 16:48:46.925140 (+ 145us) client_negotiation.cc:240] Negotiation successful
0619 16:48:46.925442 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":195}
I20260619 16:48:46.927239 23423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:48:46.896927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58898 (local address 127.0.97.60:34355)
0619 16:48:46.897098 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0619 16:48:46.897104 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:48:46.897121 (+ 17us) server_negotiation.cc:407] Connection header received
0619 16:48:46.897175 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:48:46.897180 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:48:46.897251 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:48:46.897339 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:48:46.898680 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:48:46.899787 (+ 1107us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:48:46.901865 (+ 2078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:48:46.902579 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:48:46.902716 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:48:46.918875 (+ 16159us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:48:46.918882 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:48:46.918887 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:48:46.918927 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:48:46.922690 (+ 3763us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:48:46.923626 (+ 936us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:48:46.923633 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:48:46.923637 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:48:46.923792 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:48:46.924234 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:48:46.924239 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:48:46.924243 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:48:46.924584 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:48:46.924760 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0619 16:48:46.926578 (+ 1818us) server_negotiation.cc:299] Negotiation successful
0619 16:48:46.926856 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260619 16:48:46.933461 23346 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9e1408dd44dd42c6b49493e1be8ffb36: Bootstrap starting.
I20260619 16:48:46.902468 23205 master_runner.cc:186] Error getting master registration for 127.0.97.61:36667: OK, instance_id { permanent_uuid: "9e1408dd44dd42c6b49493e1be8ffb36" instance_seqno: 1781887726174817 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260619 16:48:46.941300 23205 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260619 16:48:46.945858 23346 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9e1408dd44dd42c6b49493e1be8ffb36: Neither blocks nor log segments found. Creating new log.
I20260619 16:48:46.947050 23455 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 16:48:46.948829 23455 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:48:46.954068 23346 log.cc:824] T 00000000000000000000000000000000 P 9e1408dd44dd42c6b49493e1be8ffb36: Log is configured to *not* fsync() on all Append() calls
I20260619 16:48:46.965073 23346 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9e1408dd44dd42c6b49493e1be8ffb36: No bootstrap required, opened a new log
I20260619 16:48:46.969892 23455 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a8932a4bca264aad92e88ef5f5a577a6: Generated new TSK 0
I20260619 16:48:46.970875 23455 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 16:48:46.978461 23346 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9e1408dd44dd42c6b49493e1be8ffb36 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8932a4bca264aad92e88ef5f5a577a6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42985 } } peers { permanent_uuid: "9e1408dd44dd42c6b49493e1be8ffb36" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36667 } } peers { permanent_uuid: "bc1be13db59442abac3d356cab9f7668" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34355 } }
I20260619 16:48:46.978840 23346 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9e1408dd44dd42c6b49493e1be8ffb36 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e1408dd44dd42c6b49493e1be8ffb36, State: Initialized, Role: FOLLOWER
I20260619 16:48:46.979343 23346 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9e1408dd44dd42c6b49493e1be8ffb36 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8932a4bca264aad92e88ef5f5a577a6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42985 } } peers { permanent_uuid: "9e1408dd44dd42c6b49493e1be8ffb36" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36667 } } peers { permanent_uuid: "bc1be13db59442abac3d356cab9f7668" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34355 } }
I20260619 16:48:46.980602 23470 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9e1408dd44dd42c6b49493e1be8ffb36 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8932a4bca264aad92e88ef5f5a577a6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42985 } } peers { permanent_uuid: "9e1408dd44dd42c6b49493e1be8ffb36" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36667 } } peers { permanent_uuid: "bc1be13db59442abac3d356cab9f7668" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34355 } } }
I20260619 16:48:46.981392 23470 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9e1408dd44dd42c6b49493e1be8ffb36 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 16:48:46.981595 23346 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9e1408dd44dd42c6b49493e1be8ffb36 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:48:46.981988 23320 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9e1408dd44dd42c6b49493e1be8ffb36 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:48:46.982126 23321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a8932a4bca264aad92e88ef5f5a577a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "9e1408dd44dd42c6b49493e1be8ffb36"
I20260619 16:48:46.985720 23320 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 9e1408dd44dd42c6b49493e1be8ffb36 [term 1 FOLLOWER]: Refusing update from remote peer a8932a4bca264aad92e88ef5f5a577a6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:48:46.989454 23437 consensus_queue.cc:1035] T 00000000000000000000000000000000 P a8932a4bca264aad92e88ef5f5a577a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e1408dd44dd42c6b49493e1be8ffb36" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36667 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:48:47.029424 23321 raft_consensus.cc:2396] T 00000000000000000000000000000000 P 9e1408dd44dd42c6b49493e1be8ffb36 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate a8932a4bca264aad92e88ef5f5a577a6 for term 1 because replica has last-logged OpId of term: 1 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20260619 16:48:47.031929 23470 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9e1408dd44dd42c6b49493e1be8ffb36 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a8932a4bca264aad92e88ef5f5a577a6. Latest consensus state: current_term: 1 leader_uuid: "a8932a4bca264aad92e88ef5f5a577a6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8932a4bca264aad92e88ef5f5a577a6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42985 } } peers { permanent_uuid: "9e1408dd44dd42c6b49493e1be8ffb36" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36667 } } peers { permanent_uuid: "bc1be13db59442abac3d356cab9f7668" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34355 } } }
I20260619 16:48:47.032373 23470 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9e1408dd44dd42c6b49493e1be8ffb36 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 16:48:47.032691 23474 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9e1408dd44dd42c6b49493e1be8ffb36 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a8932a4bca264aad92e88ef5f5a577a6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8932a4bca264aad92e88ef5f5a577a6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42985 } } peers { permanent_uuid: "9e1408dd44dd42c6b49493e1be8ffb36" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36667 } } peers { permanent_uuid: "bc1be13db59442abac3d356cab9f7668" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34355 } } }
I20260619 16:48:47.033095 23474 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9e1408dd44dd42c6b49493e1be8ffb36 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 16:48:47.090055 23490 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
W20260619 16:48:47.112460 23350 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:48:47.103433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38044 (local address 127.0.97.61:36667)
0619 16:48:47.103582 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0619 16:48:47.103588 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:48:47.110341 (+ 6753us) server_negotiation.cc:407] Connection header received
0619 16:48:47.110876 (+ 535us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:48:47.110883 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:48:47.110975 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:48:47.111044 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:48:47.112219 (+ 1175us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38044: BlockingRecv error: recv got EOF from 127.0.0.1:38044 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
W20260619 16:48:47.114672 23275 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:48:47.103108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46226 (local address 127.0.97.62:42985)
0619 16:48:47.103317 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0619 16:48:47.103323 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:48:47.113080 (+ 9757us) server_negotiation.cc:407] Connection header received
0619 16:48:47.113318 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:48:47.113324 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:48:47.113420 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:48:47.113490 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:48:47.114428 (+ 938us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46226: BlockingRecv error: recv got EOF from 127.0.0.1:46226 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
W20260619 16:48:47.116542 23423 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:48:47.113488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58904 (local address 127.0.97.60:34355)
0619 16:48:47.113701 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0619 16:48:47.113707 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:48:47.114359 (+ 652us) server_negotiation.cc:407] Connection header received
0619 16:48:47.114579 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:48:47.114595 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:48:47.114681 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:48:47.114750 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:48:47.116296 (+ 1546us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58904: BlockingRecv error: recv got EOF from 127.0.0.1:58904 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
W20260619 16:48:47.123065 23350 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:48:47.121508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38046 (local address 127.0.97.61:36667)
0619 16:48:47.121703 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0619 16:48:47.121709 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:48:47.121727 (+ 18us) server_negotiation.cc:407] Connection header received
0619 16:48:47.121846 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:48:47.121850 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:48:47.121987 (+ 137us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:48:47.122092 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:48:47.122805 (+ 713us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38046: BlockingRecv error: recv got EOF from 127.0.0.1:38046 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
W20260619 16:48:47.131441 23350 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:48:47.127487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38054 (local address 127.0.97.61:36667)
0619 16:48:47.127654 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0619 16:48:47.127659 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:48:47.130208 (+ 2549us) server_negotiation.cc:407] Connection header received
0619 16:48:47.130380 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:48:47.130385 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:48:47.130466 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:48:47.130567 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:48:47.131261 (+ 694us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38054: BlockingRecv error: recv got EOF from 127.0.0.1:38054 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
W20260619 16:48:47.142202 23350 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:48:47.140162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38062 (local address 127.0.97.61:36667)
0619 16:48:47.140315 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0619 16:48:47.140320 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:48:47.140859 (+ 539us) server_negotiation.cc:407] Connection header received
0619 16:48:47.141041 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:48:47.141046 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:48:47.141139 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:48:47.141233 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:48:47.141971 (+ 738us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38062: BlockingRecv error: recv got EOF from 127.0.0.1:38062 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
W20260619 16:48:47.153514 23350 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:48:47.151520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38066 (local address 127.0.97.61:36667)
0619 16:48:47.151697 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0619 16:48:47.151703 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:48:47.152195 (+ 492us) server_negotiation.cc:407] Connection header received
0619 16:48:47.152369 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:48:47.152374 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:48:47.152460 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:48:47.152561 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:48:47.153301 (+ 740us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38066: BlockingRecv error: recv got EOF from 127.0.0.1:38066 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
W20260619 16:48:47.166661 23350 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:48:47.164632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38076 (local address 127.0.97.61:36667)
0619 16:48:47.164798 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0619 16:48:47.164804 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:48:47.165263 (+ 459us) server_negotiation.cc:407] Connection header received
0619 16:48:47.165442 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:48:47.165448 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:48:47.165546 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:48:47.165672 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:48:47.166467 (+ 795us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38076: BlockingRecv error: recv got EOF from 127.0.0.1:38076 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
W20260619 16:48:47.213375 23350 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:48:47.211382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38078 (local address 127.0.97.61:36667)
0619 16:48:47.211567 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0619 16:48:47.211575 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 16:48:47.212058 (+ 483us) server_negotiation.cc:407] Connection header received
0619 16:48:47.212252 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:48:47.212256 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:48:47.212340 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:48:47.212431 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:48:47.213166 (+ 735us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38078: BlockingRecv error: recv got EOF from 127.0.0.1:38078 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
W20260619 16:48:47.250658 23350 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:48:47.248566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38086 (local address 127.0.97.61:36667)
0619 16:48:47.248764 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0619 16:48:47.248772 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 16:48:47.249282 (+ 510us) server_negotiation.cc:407] Connection header received
0619 16:48:47.249498 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:48:47.249502 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:48:47.249575 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:48:47.249659 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:48:47.250406 (+ 747us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38086: BlockingRecv error: recv got EOF from 127.0.0.1:38086 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
W20260619 16:48:47.320829 23350 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:48:47.318386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38094 (local address 127.0.97.61:36667)
0619 16:48:47.318685 (+ 299us) server_negotiation.cc:206] Beginning negotiation
0619 16:48:47.318694 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 16:48:47.319295 (+ 601us) server_negotiation.cc:407] Connection header received
0619 16:48:47.319522 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:48:47.319529 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:48:47.319616 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:48:47.319738 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:48:47.320572 (+ 834us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38094: BlockingRecv error: recv got EOF from 127.0.0.1:38094 (error 108)
Metrics: {"server-negotiator.queue_time_us":126}
I20260619 16:48:47.434152 23437 sys_catalog.cc:455] T 00000000000000000000000000000000 P a8932a4bca264aad92e88ef5f5a577a6 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a8932a4bca264aad92e88ef5f5a577a6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8932a4bca264aad92e88ef5f5a577a6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42985 } } peers { permanent_uuid: "9e1408dd44dd42c6b49493e1be8ffb36" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36667 } } peers { permanent_uuid: "bc1be13db59442abac3d356cab9f7668" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34355 } } }
I20260619 16:48:47.434556 23437 sys_catalog.cc:458] T 00000000000000000000000000000000 P a8932a4bca264aad92e88ef5f5a577a6 [sys.catalog]: This master's current role is: LEADER
W20260619 16:48:47.453852 23350 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:48:47.451938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38110 (local address 127.0.97.61:36667)
0619 16:48:47.452098 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0619 16:48:47.452103 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:48:47.452455 (+ 352us) server_negotiation.cc:407] Connection header received
0619 16:48:47.452651 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:48:47.452655 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:48:47.452730 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:48:47.452812 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:48:47.453552 (+ 740us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38110: BlockingRecv error: recv got EOF from 127.0.0.1:38110 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
W20260619 16:48:47.717197 23350 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:48:47.715363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38114 (local address 127.0.97.61:36667)
0619 16:48:47.715583 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0619 16:48:47.715590 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:48:47.715784 (+ 194us) server_negotiation.cc:407] Connection header received
0619 16:48:47.716044 (+ 260us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:48:47.716049 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:48:47.716128 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:48:47.716226 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:48:47.717024 (+ 798us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38114: BlockingRecv error: recv got EOF from 127.0.0.1:38114 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
I20260619 16:48:47.839944 23458 catalog_manager.cc:1259] Loaded cluster ID: 4dd204c7646d460fbb4beda09a62095f
I20260619 16:48:47.840065 23458 catalog_manager.cc:1552] T 00000000000000000000000000000000 P bc1be13db59442abac3d356cab9f7668: loading cluster ID for follower catalog manager: success
I20260619 16:48:47.842555 23458 catalog_manager.cc:1574] T 00000000000000000000000000000000 P bc1be13db59442abac3d356cab9f7668: acquiring CA information for follower catalog manager: success
I20260619 16:48:47.844269 23458 catalog_manager.cc:1602] T 00000000000000000000000000000000 P bc1be13db59442abac3d356cab9f7668: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 16:48:48.094146 23490 catalog_manager.cc:1259] Loaded cluster ID: 4dd204c7646d460fbb4beda09a62095f
I20260619 16:48:48.094260 23490 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 9e1408dd44dd42c6b49493e1be8ffb36: loading cluster ID for follower catalog manager: success
I20260619 16:48:48.096728 23490 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 9e1408dd44dd42c6b49493e1be8ffb36: acquiring CA information for follower catalog manager: success
I20260619 16:48:48.098450 23490 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 9e1408dd44dd42c6b49493e1be8ffb36: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 16:48:48.241631 23554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:48:48.235479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38130 (local address 127.0.97.61:36667)
0619 16:48:48.236141 (+ 662us) server_negotiation.cc:206] Beginning negotiation
0619 16:48:48.236146 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:48:48.236164 (+ 18us) server_negotiation.cc:407] Connection header received
0619 16:48:48.236223 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:48:48.236227 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:48:48.236373 (+ 146us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:48:48.236477 (+ 104us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 16:48:48.236907 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:48:48.237704 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:48:48.239031 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:48:48.239520 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:48:48.239648 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:48:48.240208 (+ 560us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 16:48:48.240972 (+ 764us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 16:48:48.241132 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0619 16:48:48.241213 (+ 81us) server_negotiation.cc:299] Negotiation successful
0619 16:48:48.241297 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":530,"thread_start_us":381,"threads_started":1}
I20260619 16:48:48.256219 23560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:48:48.247261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58912 (local address 127.0.97.60:34355)
0619 16:48:48.250379 (+ 3118us) server_negotiation.cc:206] Beginning negotiation
0619 16:48:48.250384 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:48:48.250399 (+ 15us) server_negotiation.cc:407] Connection header received
0619 16:48:48.250499 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:48:48.250503 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:48:48.250642 (+ 139us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:48:48.250777 (+ 135us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 16:48:48.251292 (+ 515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:48:48.252034 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:48:48.253908 (+ 1874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:48:48.254385 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:48:48.254483 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:48:48.255054 (+ 571us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 16:48:48.255582 (+ 528us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 16:48:48.255724 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0619 16:48:48.255845 (+ 121us) server_negotiation.cc:299] Negotiation successful
0619 16:48:48.255928 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3025,"thread_start_us":2932,"threads_started":1}
I20260619 16:48:48.275913 23554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:48:48.262007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38132 (local address 127.0.97.61:36667)
0619 16:48:48.262181 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0619 16:48:48.262187 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:48:48.268938 (+ 6751us) server_negotiation.cc:407] Connection header received
0619 16:48:48.269216 (+ 278us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:48:48.269221 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:48:48.269299 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:48:48.269358 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 16:48:48.270291 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:48:48.271384 (+ 1093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:48:48.273558 (+ 2174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:48:48.274246 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:48:48.274326 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:48:48.275143 (+ 817us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 16:48:48.275270 (+ 127us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 16:48:48.275329 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0619 16:48:48.275543 (+ 214us) server_negotiation.cc:299] Negotiation successful
0619 16:48:48.275623 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260619 16:48:48.276464 23566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:48:48.261683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46242 (local address 127.0.97.62:42985)
0619 16:48:48.268550 (+ 6867us) server_negotiation.cc:206] Beginning negotiation
0619 16:48:48.268557 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:48:48.268939 (+ 382us) server_negotiation.cc:407] Connection header received
0619 16:48:48.269169 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:48:48.269175 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:48:48.269370 (+ 195us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:48:48.269537 (+ 167us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 16:48:48.270255 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:48:48.271332 (+ 1077us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:48:48.273558 (+ 2226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:48:48.274233 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:48:48.274327 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:48:48.275202 (+ 875us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 16:48:48.275637 (+ 435us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 16:48:48.275706 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0619 16:48:48.275905 (+ 199us) server_negotiation.cc:299] Negotiation successful
0619 16:48:48.276198 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6740,"thread_start_us":6635,"threads_started":1}
I20260619 16:48:48.280789 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu pid 23205
I20260619 16:48:48.281695 23267 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:48:48.283087 23560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:48:48.269605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58926 (local address 127.0.97.60:34355)
0619 16:48:48.269798 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0619 16:48:48.269804 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:48:48.276420 (+ 6616us) server_negotiation.cc:407] Connection header received
0619 16:48:48.276608 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:48:48.276613 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:48:48.276703 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:48:48.276786 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 16:48:48.278474 (+ 1688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:48:48.279378 (+ 904us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:48:48.281210 (+ 1832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:48:48.281800 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:48:48.281925 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:48:48.282329 (+ 404us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 16:48:48.282464 (+ 135us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 16:48:48.282551 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0619 16:48:48.282694 (+ 143us) server_negotiation.cc:299] Negotiation successful
0619 16:48:48.282782 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260619 16:48:48.407267 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu with pid 23205
I20260619 16:48:48.424441 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu pid 23279
I20260619 16:48:48.425567 23341 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:48:48.534363 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu with pid 23279
I20260619 16:48:48.550057 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu pid 23353
I20260619 16:48:48.551103 23415 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:48:48.654397 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEs2EH8/build/asan/bin/kudu with pid 23353
2026-06-19T16:48:48Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (4562 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (9146 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (342713 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST